The Great Switcheroo of Noses, Mouths, and Eyes

What swapping facial features teaches us about the importance of the eyes in face perception

When Bruce Lee fights Chuck Norris in The Way of the Dragon, you have no difficulties telling who is who – even if you are not too familiar with the looks of both individuals, you can easily recognize Bruce Lee as the actor with Asian features and Chuck Norris as his European-looking opponent. But would you get confused if Bruce Lee’s nose appeared in Chuck Norris’ face, or if Chuck Norris’ eyes were swapped for those of Bruce Lee?

Scientists have been interested for a long time in how people classify faces – for example, as male or female, or as European or Asian. Do people judge that a person looks Asian or European based on the eyes, the nose, or the mouth, or maybe rather the overall shape of the face?

Isabelle Bülthoff of the Max Planck Institute for Biological Cybernetics in Tübingen is one of the scientists looking for answers to these questions. “When asked about the most striking difference between Korean and German faces, most people would intuitively answer that it is the eyes – and we wanted to test if this is indeed true,” says Bülthoff.

An eye for an eye, a nose for a nose

To find out, Bülthoff and her collaborators from Korea University in South Korea created new faces by exchanging various facial features between pictures of Europeans and Asian. The researchers started out from real faces of 20 Germans and 20 Koreans, just like those displayed in the left column of the figure, where the upper face belongs to a German and the lower one to a Korean. To allow a better view of the shape of the nose, the cheeks, and the chin, all faces are turned a bit to the side.

One feature at a time, the scientists then modified these faces. They exchanged, for example, the eyes in a European face with Asian eyes, while the rest of the face stayed the same (see middle column, upper face), and conversely the eyes in an Asian face with European eyes, again leaving the rest of the face unchanged (middle column, lower face). The faces in the rightmost column show a different modification: the upper face is again the same German face as before, only this time with an Asian nose, and the lower image shows the same Korean face, only with the mouth replaced by the mouth of a European. Bülthoff stresses the effort that the team took to make the faces look lifelike: “We had to take care that each feature was embedded naturally into the faces – all faces look realistic, and it is very hard to tell which faces were manipulated.”

What determines how we perceive faces?

Participants from Germany were then asked if the faces – the 40 original faces as well as all their variations – looked Asian or European. Among all modifications, a change of the eyes was most impactful for perception: a majority of the German participants classified the Asian faces with European eyes as European-looking; and conversely, most classified European faces with Asian eyes as Asian-looking. This strong impact on perception was not observed when the nose or the mouth was exchanged. “Even when the whole face is European except for the eyes, its Asian eyes are sufficient to make the face Asian-looking, and vice versa,” explains Bülthoff.

But the researchers wondered if this effect might be caused by cultural factors; or maybe it could be influenced by the fact that Germans typically see more European than Asian faces in their daily lives. Therefore, they repeated the experiment in Korea. It turned out that just like their German counterparts, the Korean participants also found the eyes to be the most important facial feature for making a face Asian- or European-looking.

This suggests that across different cultures and geographic regions, the eyes play a crucial role in face perception. So while there is no way a switcheroo of the eyes would escape us, we are left to wonder if we would even notice Chuck Norris’ nose in Bruce Lee’s face.