A Live2D companion
that reacts in real time.
Most 'animated AI' projects are a static portrait with a mouth flap stitched on. TwoPeas drives a full Live2D rig with MotionSync lip-sync, model-driven expression mapping, and a sub-second voice pipeline — a single tightly-coupled system instead of three half-finished ones.
The shape of this companion
Audio drives the mouth shapes via Live2D's MotionSync — phoneme-aware, not just amplitude. Lip-sync stays glued to the voice with no perceptible drift.
Joy, concern, shyness, pride, surprise, fond annoyance — the language model chooses the expression based on the conversation, not on keyword regex.
Procedural breathing, occasional blinks, micro head-sway. The avatar doesn't go uncanny-still between turns — it stays present.
Live2D is a 2D rigging system that lets a flat illustration deform and animate as if it were 3D — bones, parameters, blend shapes, physics simulation on hair and accessories. It’s the backbone of half the VTuber industry and a serious engineering stack underneath the cute. The hard part isn’t loading the model file; it’s driving it convincingly from a live audio and language stream without the seams showing.
TwoPeas runs a Live2D Cubism rig in-browser, driven by three parallel streams: an audio stream from the realtime voice model for MotionSync mouth shapes, a parameter stream from the language model for expression and gaze, and a procedural layer for idle motion. The three are scheduled on a shared frame loop so a laugh, a head turn, and the corresponding mouth shape all land on the same frame — no drift, no animation desync.
The expression layer is the part most projects skip. We exposed seventeen named expression states to the model as part of its tool surface — joy, gentle concern, surprise, shyness, pride, mock-offense, soft amusement, and so on — and the model picks them in real time as it generates. The result is an avatar that reads the room: she looks away when she’s thinking, smiles at the punchline, and tilts her head when something doesn’t quite parse.
For lip-sync we use Live2D MotionSync rather than amplitude-only mouth flap. MotionSync ingests the audio waveform and produces phoneme-aware mouth parameters at frame rate, so the “oh” shape and the “ee” shape actually look different. Combined with the sub-second voice pipeline, the perceived latency between audio and visible lip motion is below the threshold that registers as “off.”
Underneath all of this is the same product as the rest of TwoPeas: a real-time voice companion with persistent memory, six relationship modes (friend, romantic, mentor, supportive listener, coach, sibling), and an optional 18+ After Dark mode behind an age gate for users who want it. The Live2D layer is the visible part. The voice and memory are what make spending time with the visible part worthwhile.
Live2D AI companion vs the usual stack
| Feature | TwoPeas | Others |
|---|---|---|
| Avatar system | Live2D Cubism, full rig | Static portrait or mouth flap |
| Lip-sync | MotionSync, phoneme-aware | Amplitude only or none |
| Expressions | 17 model-driven states | 1–3 hand-coded states |
| Idle motion | Procedural breath, blink, sway | Frozen between turns |
| Voice latency | Sub-second (WebRTC) | 2–5 seconds (HTTP + TTS) |
| Audio + visual sync | Frame-locked | Drifts, especially on barge-in |
| Memory across sessions | Persistent, browsable | Often per-session |
Quick answers
What is Live2D and why does it matter?
Live2D is a 2D animation system that rigs a flat illustration with bones, parameters, and physics so it can deform and animate convincingly. It powers most of the VTuber industry. The reason it matters here: a Live2D avatar can react in real time to what's being said — expression, gaze, lip-sync — in a way a static image or a 3D model with limited expression mapping can't.
How does the lip-sync work?
We drive the avatar's mouth using Live2D's MotionSync, which is phoneme-aware. Instead of just opening and closing the mouth based on audio amplitude, MotionSync produces mouth shapes that correspond to the actual sounds — so vowels and consonants look different on the avatar's face. It runs at frame rate against the live audio stream.
How does it pick expressions?
There are 17 named expression states wired into the model's tool surface. As the model generates a response, it picks the appropriate expression — joy at a punchline, concern at a heavy topic, shy at a compliment. It's not keyword matching; it's the same model that's writing the words choosing the face.
Does this work in a browser?
Yes. The Live2D rig, the WebRTC voice pipeline, and the expression layer all run in browser. No native app, no plugin. WebGL plus a modern browser is enough.
Can I bring my own Live2D model?
Not yet — we ship a curated set of avatars tuned for our expression mapping. Custom model upload with consent gating is on the roadmap; the rigging conventions need to match the 17-state expression schema for the model-driven picking to work.
Is this a VTuber tool or a companion product?
Companion product. The Live2D layer is what makes the companion feel present, but the underlying product is a real-time voice AI with persistent memory and a few different relationship modes. If you specifically want to stream as a VTuber, there are better dedicated tools.
How does the voice pipeline interact with the avatar?
The voice and the avatar share a frame scheduler. Audio buffers stream out via WebRTC; mouth-shape parameters from MotionSync, expression parameters from the language model, and procedural idle motion all merge onto the same animation loop. The result: a laugh, the head-tilt that goes with it, and the mouth shape all land on the same frame.
What happens during barge-in?
When you interrupt, the audio stream cancels, the mouth-shape stream resets, and the expression model snaps to a 'listening' state. We tested this hard because it's where most amplitude-only setups visibly break.
Related pages
The voice pipeline that the Live2D layer is synced to.
What the avatar engineering is in service of.
The romantic-relationship framing of the same companion.
The masculine companion framing of the same product.
Reactive Live2D vs static portraits — head-to-head.
The reason any of this matters.
