A photo that depicts a teenager in a red “Make America Great Again” hat smiling and standing in front of a Native American elder has circulated widely on social media and generated national outrage. The photo captures one moment in an encounter among the teenager’s high school class, a Native American group, and a religious group during the Indigenous Peoples March in Washington, D.C., on Friday.

Online, people were quick to pass judgement about the situation, and fell into two discrete camps—one that said the teenager was harassing the Native elder, and another that said the elder was harassing him. In most cases, the deciding factor was the teenager’s expression, read either as smug or respectful.

The problem is, neither reading of his face is correct because faces can’t be read.

“You can’t look at someone’s face and know how they feel. The evidence is very clear on this,” said Lisa Feldman Barrett, a University Distinguished Professor of psychology at Northeastern who studies the way humans express emotion.

Since the incident caught fire online, both the teenager and the Native American elder have relayed their sides of the story, and the archdiocese presiding over the teen’s Catholic school has condemned his behavior.

Barrett, who has delivered a TED talk on this topic, said that gauging someone’s emotional state involves taking stock of all the information available at the time, as well as an awareness of our own inherent biases.

“A face is just a character in a more extended story that is unfolding over time,” she said.

Other factors—including vocal acoustics, body posture, an understanding of what else is going on in the situation, and an appreciation of the relationship between the people involved—are all part of the context that a brain uses to guess the meaning of facial movements, Barrett said.

And even with all that context, there is still the matter of “your internal state as a perceiver,” Barrett said.

“The physical state of your body, and your beliefs, your prior experiences, those are part of the context, too,” she said.

That’s because, Barrett said, our brains don’t extract the emotional meanings in expressions as much as search back through past experiences to find a good comparison that gives meaning to the facial movements.

“The brain isn’t asking, ‘What is this expression?’ Your brain is asking, ‘What are these facial movements, as they occur in the present context, similar to, based on past experiences?’” she said. “Your brain is using information from your past experiences where you encountered someone making a similar face in similar circumstances.”

The brain does all this so quickly that inferring the emotional meaning in someone’s facial movements feels like “face reading,” when in fact “your brain is always just guessing,” Barrett said.

There’s a lot happening in the video that might elicit a response based on a viewer’s previous experiences and convictions. There are the “Make America Great Again” hats, the dynamic of a younger person making direct eye contact with an older person, and a person of lighter skin tone interacting with a person of darker skin tone. All of these are ingredients that the brain uses to guess at the meaning of the teenager’s facial movements, which are experienced as “reading a facial expression.”

“This is just how brains work,” Barrett said. “They use the past in order to make sense of the present.”

For media inquiries, please contact Shannon Nargi at s.nargi@northeastern.edu or 617-373-5718.