The eyes may be windows into the soul, but can we really garner that much information from staring into someone’s face? Research suggests maybe not.

A whole business has developed out of training computers to recognize our facial expressions, with companies like Emotient and Affectiva selling facial recognition software that supposedly reveals how focus groups respond to advertisements or how shoppers feel. Agencies like the CIA and the TSA have used the facial emotion research of psychologist Paul Ekman to try to examine the tiniest changes in expression for signs of potential deception or ill intent. Companies like Apple and Google are also working on facial recognition technology, although Google has tried to keep Glass apps facial-recognition free (for now at least).

Yet there’s a major issue with training computers (or even people) to read facial expressions to evaluate behavior: Sometimes you just can’t tell what’s going on inside someone’s head by looking at his or her face. Northeastern University professor Lisa Feldman Barrett, a psychologist who studies emotions, writes in The New York Times that her research indicates “that human facial expressions, viewed on their own, are not universally understood.”