Let’s put a smile on that face: How you can tell if you’re looking at a fake smile

Unless you’re facing someone cosplaying as the Joker, whose smile is obviously nothing more than scars and greasepaint, how can you tell if that smile is for real?

AI can tell, and even Batman’s nemesis would have a hard time lying to a computer. Now a newly upgraded AI that uses FACS, or the Facial Action Coding System, levels up previous programs designed to analyze facial expression. It can pretty much see right through you by using computer vision to distinguish between spontaneous (genuine) and forced smiles to catch many of the standout features that set them apart. This AI has been programmed to catch the telltale signs that a supervillain — or anyone else — is forcing a smile.

What a smile really is goes far beyond just a mask. While deep learning programs have been trained to discern between spontaneous and forced smiles before, the technology has now been improved to take even more specific aspects of a smile into its robo-mind before coming up with a conclusion.

“We have developed a new technique for identifying the smile phases, which is robust against the noise and allows for continuous analysis of facial videos,” the scientists said in a study recently published in PLOS ONE.

Smiles aren’t just universal symbols of happiness like the iconic yellow emoji. Someone can easily sucker you into believing them with an artificial smile. No matter how much chemical sweetener they put into it, there are ways to tell whether or not you’re looking at a fake. FACS ditches the anatomical and emotional aspects of a smile and sees the face only through the eyes of an observer. Meaning, any facial structures that are beyond the subject’s control and may influence a certain mouth position (like the infamous RBF) are out, and so are emotional factors that may influence how a smile comes across.

Whether it’s Heath Ledger’s Joker, whose father literally carved an exaggerated smile into his face, or Joaquin Phoenix’s, who paints it on in screaming red, both characters were smiling about all the wrong things. Credit: DC/Warner Bros.

While happiness can switch on a smile, so can embarrassment, nervousness, excitement, or that evil sort of grin you’d expect of the Joker when he's thinking up his next nefarious plot. This study did not venture so much into the psychological aspect of smiling, but zeroed in on how to tell whether the smile was happening on its own or if the person whose face it was on had ulterior motives.

A smile triggers muscle movements around the eyes, cheeks and other parts of the face. There are three main smile phases. When a face starts to transition from a neutral expression to a smile, that is the onset. The apex is the part we are most likely to call a smile. This phase can happen with varying intensity because of many underlying physical or psychological reasons, but those do not factor into how genuine it is. Finally, the face goes back to neutral mode in the offset. These phases do not always last for the same amount of time or manifest in exactly the same way on every individual, though some AIs had been programmed to assume that.

Also involved in a smile are AUs, or Action Units. These elements of facial expressions that are detected by FACS and are then looked into further to determine how intense they are. Though there are a total of 17 AUs involved, the two particular AUs most associated with smiling are AU6, otherwise known as cheek raiser, and AU12, or lip corner puller. There are also many more subtle components that make up AUs. What has previously been noticed across all forced smiles was that the AUs were more intense whether the smile was supposed to convey positive or negative emotions, so these are obvious signs that can call out a liar desperate to convince someone.

To set apart smiles that weren’t genuine, the AI assessed individual features that belonged to certain AUs, or AU-wise features, which also include intensity and other muscle movements. Features visible across AUs or cross-AU features, can indicate related aspects of AUs and pick out what signals depend on what movements. Both AU-wise and cross-AU features can belong to one of four stretches of time that tell the AI more about how long each of the three phases lasted and the transition from one phase to the other. While there is still uncertainty about the exact points when a smile starts and ends, the algorithm used for phase detection here can still tell the difference between smiles and their intensity while ignoring any distractions.

Making the AI even more discerning involved painstakingly running individual tests in which a certain sub-AU feature was removed. This determined whether or not that particular feature was evidence of a real or forced smile.

“Every detected smile is classified based on the features extracted from AU sequences within each individual smile phase, as well as from the entire smile cycle,” the scientists said of how their AI keeps an eye out for fakes.

Tell that to any Joker in cosplay who wants to draw a lipstick smile on your face.