Accuser: [Sternly] Have you ever run a statistical test on data from an ROI that was defined by the same data?
fMRI Researcher: [Eyes twitch nervously] No sir.
Accuser: Have you ever run a correlation between an ROI and behavioral data using the same data that defined the ROI?
fMRI Researcher: [Wipes sweat from brow, cowers] No sir. Never.
Accuser: [In a rising crescendo] HAVE YOU EVER PLOTTED YOUR DATA FROM AN ROI?
fMRI Researcher: [Breaks down in tears] YES! YES I HAVE BUT I SWEAR I DIDN'T MEAN TO HURT ANYBODY!
Audience: [snickers, jeers, and dismisses researcher's entire body of work as -- cue dramatic music --VOODOO!]
No doubt you are aware of Ed Vul's voodoo correlations paper by now. It is a useful paper in that it reminds us to keep our analyses independent. Nothing new statistically, of course, but it never hurts to be reminded. I decided I really didn't want to spend much time on the topic here on Talking Brains because it is getting plenty of attention without our help. But then someone point out this to me from Ed's chapter with Nancy Kanwisher:
The most common, most simple, and most innocuous instance of non-independence occurs when researchers simply plot (rather than test) the signal change in a set of voxels that were selected based on that same signal change. ... In general, plotting non-independent data is misleading, because the selection criteria conflate any effects that may be present in the data from those effects that could be produced by selecting noise with particular characteristics.
One might take from this that we're not supposed to (allowed to) plot data from an ROI, else the VOODOO police will hunt us down. But it would be a huge mistake not to look and not to publish these graphs. By looking at the graphs you can distinguish between differences that result from different levels of positive activation in the various conditions, different degrees of negative activation (signal decrease relative to baseline), or to one condition going positive and another going negative. These different patterns may suggest different theoretical interpretations. It could also suggest other differences such as the latency of the response that would go undetected in activation maps alone. If all we looked at were contrast maps, without examining amplitude or timecourse plots, we could miss out on important information. Just be aware that when you look at the magnitude of the amplitude differences, it could be slightly biased.
To be clear, Vul and Kanwisher will not prick their voodoo dolls for any researcher who wants to publish graphs from an ROI. In fact, they are quick to note the value of examining and publishing such graphs, as long as they are used to explore activation patterns that are orthogonal to the selection criteria:
On the other hand, plots of non-independent data sometimes contain useful information orthogonal to the selection criteria. When data are selected for an interaction, non-independent plots of the data reveal which of many possible forms the interaction takes. In the case of selected main effects, readers may be able to compare the activations to baseline and assess selectivity. In either case, there may be valuable, independent and orthogonal information that could be gleaned from the time-courses. In short, there is often information lurking in graphs of non-independent data; however, it is usually not the information that the authors of such graphs draw readers’ attention to. Thus, we are not arguing against displaying graphs that contain redundant (and perhaps biased) information, we are arguing against the implicit use of these graphs to convince readers by use of non-independent data.
Given the extensive buzz about all this voodoo, I sincerely hope that things don't degenerate into a witch hunt. Ed and colleagues have rightfully reminded us to be careful in our statistical treatment of fMRI data. But let's be equally careful not turn into mindless persecutors.
P.s., in case your appetite for voodoo commentary is yet to be satiated check out Brad Buchsbaum's recent critiques.