Effects of expertise on action understanding -- confounds in a high-profile study

A 2008 study published in Nature Neuroscience claims to show that elite basketball players can better predict the outcome of a free-throw shot compared to expert observers (coaches) or novices.  This result has been cited over 400 times and is repeatedly put forward as clear evidence of the perceptual effect of motor experience--i.e., support for the mirror neuron theory of action understanding.

The effect is pretty dramatic as the graphs here show.  The black circle curve shows the percentage of correct responses (indicating whether the shot would be successful or fail) as a function of how much of the clip was shown for players (top graph) versus the other groups (middle and bottom graphs).

Compare the black circle correct response curve to the grey square curve, (percent incorrect) for the three groups and notice (1) that players are better and (2) they are better earlier by a substantial margin.  In fact, the performance advantage for players occurs prior to the ball leaving the hand, suggesting that they are keying on some aspect of the kinematics rather than ball trajectory.

As I admitted in my recent response to Rizzolatti's critique of my book, the Myth of Mirror Neurons, this is an impressive effect, not easily brushed aside as trivial.  My explanation in that response was that players, unlike coaches, are motivated to predict missed versus made shots because they have to react on the court to a possible rebound.  Therefore, players are simply more skilled at noticing predictors of shot success because they have more perceptual experience looking for them. I figured they were reading the kinematics like the authors suggested.

I was interested enough in this demonstration to attempt a replication, which is still ongoing. What we have done so far, and what led to this post, is that we filmed an elite basketball player making dozens of free-throw shots. What we learned led me to partially rethink the source of the effect--the cue to shot success--reported in the Nature Neuroscience report.

Our plan was to film a bunch of shots, most successful and many (we thought) unsuccessful.  Then we would use these hits and misses as our stims for the replication.  After a couple of dozen successful shots in a row we (i.e., my post doc Jon Venezia and student Leon Wojno) realized that our player's hit rate was too high to get enough misses just by letting him take his best shots (it would have taken hours).  So they asked him to try to miss on occasion, which he did.  He missed left or right mostly, very much on purpose.

When they got back to lab and sorted the shots into hits and misses, they noticed that it was impossible to predict which way the shot would go until it was in the basket.  The question then was how could anyone in the published study (novices included) be as successful as they were?  My students re-examined the sample stims from the published study and realized how.  The misses are very bad misses.  Here is a screen shot from the supplementary material published with the study.

Airball.  And a pretty horrible one at that.  This means that the ball trajectory is quite a good cue to shot success or failure, making it possible for even novices to accurately predict the shot outcome.

But still, the players are predicting outcome even before the ball leaves the shooters hands.  Doesn't this mean that playing experience is translating (via mirror neuron resonance) to better action understanding (outcome prediction)? Not so fast.

Remember we learned that in filming the shooter we had to ask him to miss on purpose.  No doubt the authors of the published study had to do the same for their shooter, especially given that the misses were so dramatic.  This made me wonder whether there might be something in the shooter's body language that tipped his intention--like his eye gaze.  It's not news that sports require exquisite eye-hand coordination.  Free throw shooters are trained to look at the back of the rim (i.e., where they want the ball to hit).  Now, if a player were to try to miss, especially to miss short, he or she might instead look short.  I looked at the eye gaze in sample IN vs. OUT shots and indeed, the IN shot (right frame below) reveals a higher-angle gaze than the OUT shot (left frame below).  He appears to be looking where he wants the ball to go, just like the skilled player he is.

Here's another example from another stim provided by the authors.  Can you tell which one is going in and which is going to miss short?

Now that you know what to look for, it's easy! The left one goes in, the right one misses short.

So this seems to be a very likely explanation for what the elite players in the study keyed on in making their response decisions.  The authors went through a lot of trouble measuring joint angles and such to find the kinematic cues that the athlete's mirror system resonated with, but they never looked at eye gaze (or face angle).  I suspect that it would be a reliable cue.

Do athletes pay attention to eye gaze?  OF COURSE! This is how you read the intentions of the players who have the ball and why Magic Johnson's no-look pass was so effective.

So what does this high-profile paper show?  That players are better at using eye-gaze information to predict shot outcome in a situation when the shooter is trying to miss. This is not a mirror resonance effect.  It is an effect of perceptual experience that comes from learning contingencies between (sometimes very brief) eye-gaze information and action intentions.  It has nothing to do with the observers' mirroring the shot itself.