Diffusion Models & You
What’s in the box?
Build note: this is the densest guide in the section and two of three widgets want WebGL (three.js). Natural phasing line — ship the fold + scrubber first, ship the model cloud after as a headline upgrade. The three.js dependency needs a sign-off before it goes in.
A diffusion model isn’t a generator. It’s a landscape of learned concepts, and image generation is a guided walk through it. Here’s the landscape, the space it lives in, and how the walk actually happens.
[Cold open. Inference-side only, for the working artist. Anchor in canonical UNet diffusion; one line acknowledging that newer architectures (DiT, flow matching) shift internals but preserve the three-piece structure.]
The model is a landscape
[What the model is: a learned concept landscape. Concept regions and overlap; how blending works; what CFG actually does to a sample’s trajectory. LoRAs as small, localized, additive deformations of the landscape.]
CFG decides how hard you commit to a region; a LoRA reshapes just one corner of it.
Where the model works
[Latent space as the model’s native working space; the VAE as translator. The two-part aha: the model never works on “the image,” it works on a 64×64×4 latent stack; and when the VAE decodes back, it’s inventing a plausible 512×512×3 from a hint. Encoder throws away ~94% of spatial pixels; decoder makes them up at the end.]
The model lives in latent space; the VAE is the only thing that speaks both latents and pixels.
A 2D RGB image animates into a 64×64×4 latent stack — spatial compression 8× per dimension, channels expanded from 3 perceptual to 4 learned. Toggle which of the 4 channels you’re viewing; each looks vaguely like the original in a weird palette, because the channels aren’t color, they’re learned features.
The denoising loop
[What the model does over time: the denoising loop as a path through latent space, guided by text-encoder pulls. Most of what matters happens early — which is why low-step samplers work surprisingly well, and practical intuition for when to bail on a generation.]
Denoising front-loads the decisions: the picture is mostly decided long before it looks finished.
A pre-generated sequence of 20–30 denoising steps, noise → image. Drag the slider: the image scrubs, a small latent-space inset scrubs alongside at 1/8 res, and a readout calls out what each phase is doing (composition early, forms and lighting mid, texture late).
Source frames for the scrubber: a single generation captured at every step from pure noise to final image, plus the matching latent-channel previews.
The three pieces
[Tie it together: model / text encoder / VAE, each named with the role it plays. The structure that survives across architectures.]
The UNet denoises, the text encoder steers, the VAE translates. Three pieces, every model.
What’s not in here
[Explicitly out of scope, each its own future guide: training, prompt engineering, specific model comparisons (stale in three weeks), ControlNets, LoRAs-as-tutorial, IPAdapters, img2img mechanics.]
[Closer line. To be written.]
Glossary
Every term used in this guide, defined once. In the prose, underlined-dotted terms show their short definition on hover or focus, and jump here on click.
- Latent space
- The compressed space the model actually operates in — not pixels. A 512×512×3 image lives here as roughly 64×64×4 learned features.
- VAE
- Variational autoencoder. Encodes an image into latent space and decodes latents back into pixels — inventing a plausible full-res image from a low-res hint.
- CFG
- A dial on how hard the sample is pulled toward the prompt. Low CFG leaves the result ambiguous, in the overlap between concepts; high CFG commits hard to one.
- LoRA
- A small, additive deformation of a localized region of the model. A few megabytes can transform output because it edits one neighborhood, not the whole landscape.
- Denoising
- The iterative loop that walks from pure noise to a finished image. Composition locks in early; texture and detail refine late.
- Text encoder
- The component that converts your prompt into the conditioning that steers the denoising path through latent space.
- UNet
- The network that predicts and removes noise at each step. Newer architectures (DiT, flow matching) change the internals but keep the three-piece structure.