Tuesday, 4 June 2013


With greater-powered studies

An unsettling meta-analysis published by Nature –

This is currently exciting online attention from those more statistically competent than I. Here are some extracts from a straightforward and non-contentious example of this –
...the majority of neuroscience studies involve woefully small sample sizes, rendering their results highly unreliable. 'Low statistical power is an endemic problem in neuroscience,' they write.
...a comprehensive analysis of 49 neuroscience meta-analyses published in 2011 (that's all the meta-analyses published that year that contained the information required for their purposes). This took in 730 individual papers, including genetic studies, drug research and papers on brain abnormalities.
Meta-analyses collate all the findings in a given field as a way to provide the most accurate estimate possible about the size of any relevant effects... the researchers' estimate is that the median statistical power of a neuroscience study is 21 per cent. This means that the vast majority (around 79 per cent) of real effects in brain science are likely being missed. More worrying still, when underpowered studies do uncover a significant result, the lack of power means the chances are increased that the finding is spurious. Thirdly, significant effect sizes uncovered by underpowered studies tend to be overestimates of the true effect size, even when the reported effect is in fact real. This is because, by their very nature, underpowered studies are only likely to turn up significant results in data where the effect size happens to be large.
It gets more worrying...
The prevalence of inadequately powered studies in neuroscience is all the more disconcerting ...because most of the low-lying fruit in brain science has already been picked. Today, the discipline is largely on the search for more subtle effects, and for this mission, suitable studies need to be as highly powered as possible. Yet sample sizes have stood still, while at the same time it has become easier than ever to run repeated, varied analyses on the same data, until a seemingly positive result crops up.

This leads to a 'disquieting conclusion', the researchers said, 'a dramatic increase in the likelihood that statistically significant findings are spurious'...
For less non-contentious considerations of this meta-review, search on line.

Early days

Lovely sentence that: 'most of the low-lying fruit in brain science has already been picked', the applicability of which is hardly specific to neuroscience...

No comments:

Post a Comment