A talking, stylized avatar that renders in your browser — on desktop, iOS and Android

The shape of the project

One demo, three repositories

The visible artifact is a single static web page. Behind it sit three codebases that each solved a different hard problem: how to train the look, how to ship it in a browser, and how to make it fast enough for a phone.

The pipeline, end to end

Procrustes alignment to a canonical template sits between steps 2 and 3, so a face at any angle/scale maps into the framing the model was trained on.

Repo 1 · pytorch-CycleGAN-and-pix2pix

Training the look — and distilling it small

The model maps a rendered landmark image → a GTA-IV-illustrated, cyberpunk-lit face. There were no off-the-shelf labels for that, so most of the work was manufacturing the training distribution, then compressing a big teacher into a phone-sized student without losing it.

1 · From "avatar→webcam" to "landmarks→face"

The task was re-framed early: instead of mapping a specific avatar to a specific face, the input became a rendered MediaPipe FaceMesh (468 points + iris = 478) drawn in a deliberately "fancy" style — filled face oval, eyes, lips, tessellation, contours, iris circles — and the target is the masked stylized face. Keying off landmarks makes the model robust to any face. To accept landmarks from any camera, a 3D Procrustes similarity transform maps detected points onto a canonical template, removing pose, scale and translation before the GAN ever sees them.

2 · Stylization: Stable Diffusion, stacked

The face targets were stylized into a GTA-IV illustrated look with SD 1.5 + ControlNet-canny + IP-Adapter-FaceID + a GTA LoRA, in a two-pass img2img refine so the LoRAs could stack without collapsing identity. (IP-Adapter-FaceID needs its negative+positive embeds stacked, torch.cat([neg, pos]) → shape [2,1,512] — a small detail that cost real time.)

3 · The relighting breakthrough — IC-Light FBC

Lighting LoRAs never produced the dramatic chiaroscuro the reference had. The fix was IC-Light — not a LoRA but a UNet replacement taking extra latent channels (8 for foreground-conditioned, 12 for foreground+background). Two findings made it usable for a dataset:

• Text→image relighting destroys identity (the face is regenerated). An img2img init ≈ 0.5 preserves identity while still relighting.
• Background-conditioned (FBC) with one fixed background image gives consistent lighting across every subject — exactly the property a coherent training set needs.

4 · Synthesizing the data the recordings lacked

The ~275 real recordings barely covered head turns or speaking mouths. Rather than collect more, the missing distribution was synthesized with LivePortrait, driven parametrically — a pitch×yaw×roll grid of implicit-keypoint transforms with no driving video. Animating a few curated stylized heroes across the grid yields many identity-, style- and lighting-consistent frames; landmarks are re-derived per frame for clean pairs. The |yaw|=30° profiles were held out as a pose-extreme test split.

5 · Distillation — 217M teacher → 2.45M student

Real-time in-browser inference needs a tiny model. The teacher (unet_256, up to ngf=128, 217M params at 512px) is frozen and a small mobile_unet_256 student is trained to mimic it — no new labels, the targets are the teacher's outputs, loss is L1 + VGG perceptual. The student architecture is deliberately deployment-shaped: Upsample+Conv instead of ConvTranspose (no checkerboarding, INT8-friendly), post-activation [Conv, BN, ReLU], and FloatFunctional skip-concats so the graph fuses cleanly on export.

6 · The resolution lesson

The single biggest GAN insight: a UNet trained at 512 but run at 256 is badly blurry — and not because of the lower resolution itself. A fully-convolutional UNet is resolution-flexible but not resolution-invariant: its BatchNorm statistics, receptive-field scale and decoder are tuned to 512-pixel spacing, so at 256 it synthesizes low-frequency mush. Distilling natively at 256 removed the mismatch — measured ~5–6× sharper (Laplacian variance), beating even a clean mipmap of the 512 output. A later check found the all-256 pipeline landed within 3.8% of the 512-teacher pipeline at one-third the training cost: student capacity, not teacher resolution, is the ceiling.

7 · Making the mouth read — v4 → v5 → v6

Speaking mouths were the last weak spot, because the pose synthesis had frozen expression. The fix arrived in three steps, and it's a clean demonstration of where a fix has to land:

1. v4 — mouth data. Transplant viseme expression-deltas (LivePortrait open_lip/laugh, later talking.pkl real speech selected by farthest-point sampling) onto each hero. The teacher learned mouths almost perfectly (0.0070) but the student barely moved (0.0411) — a distillation capacity limit, not a data limit.
2. v5 — mouth-weighted loss. A pose-aligned mouth mask weights lip pixels 5× in the distillation L1/VGG → 0.0286 (−31%).
3. v6 — GAN-free fine-tune. Warm-start from v5's best epoch, discriminator off → 0.0234 (−44% vs v3).

Model lineage

LPIPS vs target, lower = better. Pose-extreme dropped v2 0.153 → ngf24 0.0467 (teacher ceiling 0.037); mouth-region v3 0.0416 → v6 0.0234 (teacher ceiling 0.0070).

Repo 2 · RTUKChatBot

Shipping it in the browser

A single static HTML page, no build step. The interesting work was getting a quantized CNN to run correctly and fast across three execution backends, keeping a per-frame loop allocation-free, and layering a believable animation rig on top.

Execution-provider strategy: WebNN → WebGPU → WASM

On startup the page detects which backends the browser actually exposes and tries them best-first:

• WebNN (navigator.ml) maps ops to CoreML on Apple and DirectML on Windows. For a UNet-style CNN this is typically 3–10× faster than WebGPU because CoreML can route eligible ops to the Neural Engine.
• WebGPU (navigator.gpu) is the universal fallback, configured powerPreference:"high-performance" to pick the discrete GPU.
• WASM (multi-threaded) is where quantized models run — for a reason worth its own callout.

The output upscaler

A 256² generator output benefits enormously from a good upscale. The chain went bicubic → a B-spline 4-tap filter with linear-light filtering and CAS-style adaptive sharpening — sharpen in linear light, strong on skin, gentle near edges.

Resolution, by platform

Once the resolution lesson was internalized, the app ships two natively-trained models: 256 (the mobile default — bandwidth + perf) and a higher-quality 512 (desktop default; ?res=512 opt-in on mobile). Both run at the resolution they were trained at — no mismatch blur.

Cross-origin isolation for threaded WASM

Multi-threaded WASM needs SharedArrayBuffer, which needs crossOriginIsolated == true. The headers set COOP: same-origin + COEP: credentialless. credentialless (rather than require-corp) was the key choice: MediaPipe's .task model is served without a CORP header and would be blocked by require-corp; credentialless allows no-credential cross-origin loads while still unlocking threads. Browsers without it (Safari < 17.4) degrade gracefully to single-threaded.

The animation rig — additive, in canonical space

Beyond raw webcam tracking, the landmarks can be driven procedurally: automatic blink (~15/min), idle head-sway, segmented emotion presets, and lip-sync (from an audio file, the mic, or typed text spoken by TTS). Every channel is additive and applied in canonical template space after Procrustes alignment.

The chat avatar

A later Chat tab turns the face into a local-first conversational avatar: RAG (BM25 + MiniLM embeddings) feeds an on-device LLM (WebLLM), the reply is spoken by TTS, and the speech audio simultaneously drives the mouth — each TTS WAV is fed to an analyser that maps RMS→jaw-open while the text schedules the viseme shapes. Speech-to-text is Whisper (or the native API). The whole thing is DOM-decoupled so it bolts onto the existing render loop without touching the original lab tool. Nothing is sent to a server.

The hard part · iOS & Android

Making it run on a phone

"Works on my desktop" is the easy 80%. Mobile browsers — iOS Safari especially — impose a tight memory budget, gate audio behind user gestures, and vary wildly in GPU throughput. Most of the platform-specific engineering lives here.

Device-tiered frame rate & the temporal weave

Mobile runs on tiers — weak → 10 fps, mid → 20, strong → 40 — all on the single 256 model, defaulting to a fixed safe MID that real mid-range phones sustain. A measured auto-tune (measure real per-frame cost, then step down on sustained shortfall) is opt-in, because measuring during init can mis-read a capable device. A separate temporal weave path runs four distilled 128×128 generators round-robin (one per frame), weaving each into a 2×2 sub-pixel phase of a persistent 256 buffer — temporal supersampling at roughly ¼ the per-frame GAN cost.

Repo 3 · nativeGAN

An in-house int8 engine — Rust + wgpu

onnxruntime-web is a big dependency, and its WebGPU EP couldn't run the quantized model. So the inference engine was rebuilt from scratch: parse the ONNX graph, run it as hand-written WGSL compute shaders, record the whole forward pass into a single command buffer, submit once per frame. The same engine runs on Windows native and in the browser.

"The native way" — bypass ORT, parse the ONNX into wgpu

An offline tool (export_weights.py) turns an int8 QDQ ONNX file into a flat model.json + weights.bin. The engine reads that and builds a GPU program: int8 convolution over packed activations/weights (a packed 4×int8 dot product), inputs pre-quantized once, liveness-based buffer reuse, and bind groups built once and recorded cheaply each frame. WGSL is shared verbatim between the native and web builds.

If you take six things away

Key learnings

---

Two further notes that cost real time: the RTX Blackwell (sm_120) needs PyTorch cu128 — cu121 builds fail silently, so an entire model version trained on the wrong GPU before anyone noticed. And a tiny student is easily data-loading bound: pin_memory + persistent_workers + an in-RAM cache cut epoch time 36 s → 12 s.