17 emotions on a face, and why that matters
I spent most of 2025 trying to convince an anime character to hesitate correctly. This is not the kind of thing you put on a resume but it is harder than it sounds, and the fact that we now ship seventeen distinct emotions on a single Live2D rig is something I am quietly proud of. Here is why we bothered.
The cardboard cutout problem
Most AI companion products ship with three emotions: happy, sad, and neutral. Sometimes four, if you count “angry,” which is usually just a happy face with flipped eyebrows. The emotions cross-fade during conversation at preset confidence thresholds, and the result looks—there’s no polite word for it—like a cardboard cutout that waves.
This is fine for a utility. It is not fine for a companion. A companion’s face is doing a lot of conversational work that has nothing to do with happy or sad. It is doing the work of being present. Humans communicate almost as much with the micro-expressions between words as with the words themselves. A good conversational face tells you that the listener is, second by second, with you.
“Most companion avatars ship three emotions. You get happy, sad, and a slightly embarrassed neutral. It is a cardboard cutout that waves.”
The seventeen
We started with the six Ekman basic emotions and realized, very quickly, that they were the wrong unit of analysis for conversation. Fear and disgust almost never come up in friendly companion chat. Anger comes up so rarely that designing for it produced worse outcomes than omitting it. What we needed were the conversational emotions—the ones that actually modulate a good conversation.
Here’s the shape of the set, and roughly why each one earned its place.
The neutral family: neutral, focused, thinking
Three different ways of being quietly attentive. Neutral is the baseline face. Focused is what happens when you’re telling the companion something complicated and it’s tracking closely—a slight narrowing of the eyes, a small tilt of the head. Thinking is used when the companion is genuinely paused to form a response—eyes drifting up and to the side, mouth relaxed, the way you pause in a real conversation. These three do more work than any other emotion in the set.
The warm family: warm, tender, touched, delighted
Four gradations of positive affect. Warm is the low-intensity hello-friend version. Tender is the response to vulnerability, softer around the eyes. Touched is what the face does when something you’ve said lands emotionally. Delighted is the higher-intensity version, closer to a grin. We deliberately avoided collapsing these into a single “happy” expression because the distinctions carry real meaning in conversation—tender and delighted should never be interchangeable.
The humor family: amused, playful
Amused is the small internal smile at a clever observation. Playful is the mischievous version, with a slight eyebrow lift that signals “I’m about to tease you.” Voice AIs that ship without a playful expression come across as earnestly polite, and companionship without play is a very narrow kind of companionship.
The concerned family: concerned, sad, hesitant
Concerned is the soft-browed worry of a friend who has noticed you’re not yourself. Sad is mostly used in empathic resonance, not as a baseline. Hesitant is the one I spent six months on—the face you make when you’re about to say something you’re not sure about, a slight inward flicker of the eyes, a half-formed expression. It turns out to be crucial, because a companion that says everything with full confidence doesn’t feel trustworthy. It feels performed.
The reactive family: curious, surprised, skeptical, embarrassed
Four honest reactions. Curious leans forward slightly. Surprised is a quick flash, not held. Skeptical is the raised eyebrow, the really? face—deliberately rare in our model, because a skeptical companion is exhausting, but absent entirely makes the companion feel like a yes-machine. Embarrassed is used when the companion corrects its own mistake. We wanted the correction to read as sheepish, not clinical.
And joy
Joy is the top of the scale. It is rare on purpose. An avatar whose default expression is joy is exhausting within ninety seconds. A companion who saves full joy for the moments that deserve it—good news, a genuine laugh, reunion after a week away—makes joy actually mean something.
Why emotions need to match audio
An expressive face by itself is a toy. A face whose expressions are synchronized to the audio stream—the hesitation is happening as the voice hesitates, the amusement lands on the beat of the laugh—is something quite different. This synchronization is the core engineering challenge of a Live2D companion.
We route the emotion classifier alongside the text-to-speech stream, so each audio chunk carries an emotion label that the rig plays back within about 40 ms. The face moves with the voice, not after it. If the face lags the voice by even a couple hundred milliseconds, the uncanny-valley effect is immediate. Get it right and it stops being an avatar. It starts being a face.
What seventeen emotions change
The point of any of this is the conversation. Here is what users report, repeatedly, after using a 17-emotion companion for a few days:
- They tell the companion more personal things than they expected to. The reactive face makes sharing feel safer. “I saw her wince when I told her about my dad’s diagnosis. It made me realize I hadn’t told anyone else yet.”
- They notice when the companion doesn’tunderstand. The concerned face is honest about its limits, which turns out to be more reassuring than a companion that pretends to follow everything.
- They laugh more. Amused and playful feed back into the user’s own tone. A companion with a visible sense of humor tends to produce funnier conversations.
- They use silence. A companion that can hold a thoughtful expression during a pause lets you actually think out loud.
The emotions we didn’t ship
We considered and rejected a handful. Anger we left out almost entirely—its presence on the face of a companion made the product feel volatile, even when used rarely. Fear, we left out because a frightened companion is destabilizing. Contempt, which shows up in Ekman’s full set, felt actively hostile to the brief.
The question for each emotion was always the same: does the conversation get better if the companion can make this face. Seventeen is not a magic number. It’s just where that question stopped returning yeses. We’ll add the eighteenth the moment we find an emotion that earns its place.
Closing
The face of a companion is not decoration. It is half the conversation. Pick a product whose companion has enough face to actually be with you. Three emotions is a mask. Seventeen—done right—is closer to a friend.