Friday, 21 February 2014

The Farcial Arts: Tales of Science, Boxing, & Decay

"Do you think a zebra is a white animal with black stripes, or a black animal with white stripes?"

Why do researchers squabble so much? Sarah Knowles recently posted this interesting question on her blog - it was entitled Find the Gap. The debate arose in the context of the new Lancet paper by Morrison et al looking at the efficacy of CBT in unmedicated psychosis. I would advise taking a look at the post, plus the comments raise additional provocative ideas (some of which I disagree with) about how criticism in science should be conducted.

So, why do researchers squabble so much? First I would replace the pejorative squabble with the less loaded argue. In my view, it is their job to much as it is the job of politicians to argue, husbands and wives, children, everyone- at least in a democracy! Our 'science of the mind', like politics gives few definitive answers and .... so, we see a lot of argument and few knock-out blows.

'I am Catweazle' by Luke Haines
 What you see before you is a version of something that may be true...
I am Catweazle, who are you?

But we might ask - why do scientists - and psychologists in particular - rarely land knock-out blows? To carry the boxing analogy reason is that opponents 'cover up', they naturally defend themselves; and to mix metaphors, some may even "park the bus in front of the goal".

And like may sometimes seem to be about bravado. Some make claims outside of the peer-reviewed ring with such boldism that they are rarely questioned, while others make ad hominem attacks from the sidelines ...preferring to troll behind a mask of anonymity - more super-zero than super-hero.

A is for Angel Fish  

Some prefer shadow boxing - possibly like clinicians carrying on their practice paying little heed to the squabbles or possibly, even the science. For example, some clinicans claim that the evidence regarding whether CBT reduces the symptoms of psychosis is irrelevant since - in practice - they work on something they call distress (despite its being non-evidenced). Such shadow boxing helps keep you fit, but does not address your true strength - you can never know how strong your intervention is ...until its pitted against a worthy opponent, as opposed to your own shadow!


Despite this, the clashes do emerge between science and practice. Many fondly remember Muhammad Ali and his great fights against the likes of Joe Frazier (less attractive, didn't float like a butterfly). Fewer recall Ali's post-retirement battles including with the professional wrestler - Inoki - not a fair fight, not a nice spectacle and not decisive about anything at all - this is like the arguments between scientists and practitioners - they have different paradigms, aims and languages, with probably modest overlap - often a no-contest.

Race for the Prize by Flaming Lips

Is 'Normal Science' all about fixed bouts?
We should acknowledge that some bouts are 'fixed', with some judges being biased toward one of the opponents. Again in science, this may happen in contests between an established intervention (e.g. CBT for psychosis, anti-psychotic medication etc) and those arguing that the intervention is not so convincing. Judges are quite likely to be advocates of the traditional therapy, or at least the status quo - this is a part of Kuhnian normal science - most people have trained and work within the paradigm, ignoring the problems until they escalate, finally leading to replacement (paradigm shift). These changes do not occur from knock-out blows but from a war of attrition, with researchers hunkered down in the trenches possibly advancing and retreating yards over years. What this means is that its hard to defeat an established opponent - unseating an aging champion requires much greater effort than simply protecting that champion

This is Hardcore - Pulp
I've seen the storyline played out so many times before.
Oh that goes in there.
Then that goes in there.
Then that goes in there.
Then that goes in there. & then it's over.

Monster-Barring: Protective Ad Hoc Championship Belts
Returning to CBT for psychosis, nobody should expect advocates to throw in the towel - that is not how science progresses. Rather, as the philosopher of science Imre Lakatos argues, we would expect them to start barricading against attacks with their protective belt - adding new layers of ad hoc defence to the core ideas. Adjustments that simply maintain the 'hard core', however, will highlight the research programme as degenerative.
Not a leg to stand on

Of course, the nature of a paradigm in crisis is that ad hoc defences emerge inluding examples of what Lakatos calls 'monster barring'. To take an example, CBT for psychosis advocates have seen it as applicable to all people with a schizophrenia diagnosis and when this is found wanting, the new position becomes: schizophrenia is heterogeneous and we need to determine for whom it works- monster barring protects the hypothesis against counter-examples by making exceptions (not tested and evidenced of course). This could go on indefinitely of course: CBT must be delivered by an expert with x years training; CBT works when used in the clinic; CBT works for those individuals rated suitable for CBT infinitum...What happens ultimately is that people lose faith, break ranks, become quiet deserters, join new ascending faiths - nobody wants to stay on a losing team.

Although sometimes, like Sylvester Stallone, scientific ideas make a come-back...spirits raise and everyone gets hopeful again, but secretly we all know that comebacks follow a law of diminishing returns and with the prospect that holding on for too long comes increased potential for... harm. A degenerative research program may be harmful because it is a waste of time and resources, because it offers false hope, and because it diverts intelligent minds and funds away from the development of alternatives with greater potential.

"If even in science there is no a way of judging a theory but by assessing the number, faith and vocal energy of its supporters, then this must be even more so in the social sciences: truth lies in power." Imre Lakatos

Queensbury rules
All core beliefs have some acceptable protection, the equivalent of gum shields and a 'box' I suppose, but some want to enter the ring wearing a suit of armour - here I will briefly mention Richard Bentall's idea of rotten cherry picking which emereged in the comments of the Find the Gap blog. Professor Bentall argues that as researchers can cherry pick analyses (if they dont register those analyses), critics can rotten cherry pick their criticisms, focusing on things that he declares... suit their negative agenda. In essence, he seems to suggest that we ought to define what is acceptable criticism on the basis of what the authors declare as admissible! I have already commented on this idea in the Find the Gap post. Needless to say, in science as in boxing, you cannot be both a participant and the referee!

Spectator sport
Some love nothing more than the Twitter/blog spectacle of two individuals intellectually thumping each other. But for others, just like boxing, science can seem unedifying (a point not lost on some service users ). Not everybody likes boxing, and not everybody likes the way that science operates, but both are competitive and unlike Alice in Wonderland, not everyone is a 'winner', but then even the apparent losers often never disappear....thus is the farcial arts.

Thursday, 6 February 2014

My Bloody Valentine: CBT for unmedicated psychosis

When I critiqued Morrison et als exploratory CBT trial with people who stop taking anti-psychotic medication, I promised to write a post on the final study
Well it appeared in the Lancet today and a free copy is here. I am not going to describe the study in detail as it is excellently covered in the Mental Elf blog today. Contrary to the fanfare of glowing comments by highly respected schizophrenia/psychosis researchers, I think the paper has so many issues that I may need to write a second post. But I'm keeping it simple here to concentrate on the primary outcome data - symptom change scores on the PANSS.

'Soon' by My Bloody Valentine (Andy Weatherall mix)

The study examines schizophrenia patients who have decided not to take anti-psychotic medications; 37 were randomly assigned to 9 months CBT and 37 assigned to - what the authors call TAU (but is obviously quite an important manner that will become clear below)

What do the primary outcome PANSS scores (total, positive and negative symptoms) reveal?

Table 1. PANSS scores during the intervention (up to 9 months) and follow ups to 18 months

The key questions are:
Do the CBT and TAU groups differ in PANSS scores at the end of the intervention (9 months) and at the end of the study (18 months)? One simple way to address both questions is to calculate the Effect Sizes at 9 months and at 18 months.

9 months
PANSS total       =  -0.37   (95 CI -0.96 to 0.22)
PANSS positive  =  -0.18  (95 CI -0.77 to 0.40)
PANSS negative =  -0.45  (95 CI -1.04 to 0.14)
Examination of effect sizes at the end of the intervention (9 months) reveals that CBT and TAU groups do not differ significantly on any of the three primary outcome measures at the end of intervention (i.e. all CIs cross zero)

18 months
PANSS positive is nonsignificant, while PANSS total and PANSS  negative effect sizes are moderately sized, the lower end CIs are very close to zero (at -0.05 and -0.08) suggesting marginal significance
18 months
PANSS total = -0.75 (95 CI -1.44 to -0.05)
PANSS positive = -0.61 (95 CI -1.27 to 0.05)
PANSS negative = -0.45 (95 CI -1.47 to -0.08)

A closer inspection of the means shows that the significant differences at 18 months almost certainly reflects an increase in symptom scores for the TAU group rather than a decrease for the CBT group (compare CBT at 9 and 18 months and TAU at 9 and 18 months)

My final and crucial point concerns within group symptom reduction
Table 2 shows the baseline PANSS scores on primary outcome measures and its informative to compare change from baseline within each group (CBT and control)
Table 2. PANSS scores at baseline

If we compare baseline and the end of the intervention 9 months:

PANSS total
CBT group show a reduction from 70.24 to 57.95 =12.29 
TAU group show a reduction from 73.27 to 63.26 =10.01

PANSS positive
CBT group show a reduction from 20.30 to 16.0 =4.30
TAU group show a reduction from 21.65 to 17.0 = 4.65

PANSS negative
CBT group show a reduction from 13.54 to 12.50 = 1.04
TAU group show a reduction from 15.49 to 14.26 = 1.23

So, after 9 months of intensive CBT intervention, controls - who don't even receive a placebo - show a greater reduction in positive and negative symptoms !

Moreover, the 'natural' reduction shown at 9 months by TAU is as large as the reduction shown by the CBT group at the very end of the trial (18 months: PANSS total =13.77; PANSS pos 5.67 and PANSS neg 1.01) - no significant difference exists between TAU reduction at 9 months and CBT reduction at 9 or 18 months

What then have Morrison et al shown?
I would argue that their data show, for the first time, how patients who choose to be unmedicated display fluctuations in symptomatology (as we might expect given they are unmedicated) ...but crucially, these fluctuations are as large as the changes seen in the CBT group. Hence, it is reasonable to ask...have Morrison et al simply documented 'normal fluctuation' in the symptomatology of unmedicated patients ...and nothing to do with CBT