Why AI background removers fail on flat-colour art
7 min read
On this page
A sprite sheet comes back from an AI background remover with three problems, every time:
- a thin outline that used to be one pixel is now half a pixel, or gone;
- the silhouette has moved inward by a pixel or two, so the asset no longer lines up with the sprite it was cut from;
- the edge is binary — fully opaque, then fully transparent, with nothing in between.
None of these are bugs in a particular product. They are what happens when you answer the wrong question.
Two different questions
Background removal sounds like one task. It is really two:
- Segmentation — which pixels belong to the subject?
- Matting — for each edge pixel, what fraction of its colour came from the subject, and what fraction came from the background?
A neural network trained for segmentation outputs a probability map. Threshold it and you get a mask. That mask is an answer to question one. But the pixels along an anti-aliased edge are a mixture of two colours — the subject's and the background's — and a probability is not a mixture ratio.
This distinction is easy to skip over, because on photographs it barely matters. Photos have texture, depth of field, and noise; a one-pixel error hides inside the grain. Flat-colour art has none of that. A vector-style sprite is large areas of constant colour separated by crisp, mathematically clean edges. Every pixel of error is visible, and there is no texture to hide it in.
The training data makes it worse. Segmentation models learn from photographs, where subjects are usually in focus against a distinguishable background. A flat icon with a 1px dark outline on a saturated magenta field is not a hard case perceptually — it is a hard case statistically, because it barely appears in the distribution.
What a hard mask does to a one-pixel outline
Take a sprite with a 1px dark outline on a green screen. Along that outline, each pixel is roughly half subject, half background. Its true alpha is around 0.5, and its colour is a blend.
A segmentation mask has two options for that pixel: inside, or outside. Pick outside and the outline disappears. Pick inside and the outline stays, but its colour is now the blend — green-contaminated — and it sits on a hard edge with no transition.
There is no third option, because the mask has no way to express "60% of this pixel".
This is measurable. Walk a line across the silhouette of a cutout and record the alpha values you cross. On a properly matted edge you see a ramp: 255, 214, 160, 92, 31, 0. On a thresholded mask you see a cliff: 255, 255, 0, 0.
When we instrumented our own pipeline for this, the number was uncomfortable: across six representative images, 49%–52% of silhouette crossings were a direct α:255 → 0 transition — half of every edge had no anti-aliasing at all. That was a geometry-based pipeline, not a neural one. The failure mode is not specific to AI; it is specific to any approach that treats the edge as a boundary instead of a mixture.
The alpha trap we fell into ourselves
It would be convenient to end the story with "neural networks bad, colour maths good". That is not what the measurements said.
Our first alpha estimator looked reasonable:
α = (d − threshold) / (maxDist · fgRamp − threshold)
d = Euclidean RGB distance between the pixel and the screen colour
maxDist = the largest colour distance found anywhere in the image
The problem is maxDist. It is a global quantity, decided by the most saturated object in the frame. Add one vivid element and the ramp rescales for every other pixel. A wide transition band drifts from a few pixels to twenty or thirty.
Two consequences followed, both measured:
- Thick dark outlines fell into the middle of the ramp. A 20–30px dark outline was classified as a soft gradient edge and assigned
α ≈ 0.74— a solid stroke turned into a semi-transparent one. - Any solid colour not far enough from the screen colour went translucent. The dark green wall of a glass sphere came out at
α ≈ 0.42.
Then came the second half of the trap. Unmixing is a division:
C_fg = (C_observed − (1 − α) · C_bg) / α
If alpha is underestimated, the numerator goes negative and the channel clamps at zero. For one dark-red outline:
C_g = (52 − 0.26 × 249) / 0.74 = −17 → 0
A dark red stroke turned bright red. Dark colours turned black. Grey fringes appeared where the division overshot in the other direction. All of it traced back to a single number being 0.26 too low.
The lesson is not "use a better formula". It is that alpha and colour are coupled: you cannot estimate one without constraining the other. A pipeline that solves alpha first and colour second will always produce colours that don't exist in the source image.
Why more resolution does not fix it
The intuitive fix is to work at a higher resolution and downsample. It helps a little, then stops.
Doubling the resolution gives the model more pixels, but the error is not sampling error — it is a boundary decision, and the decision is still binary. A 2× mask downsampled to 1× produces a two-pixel-wide ramp, which is better than a cliff but still not the true coverage. Meanwhile the cost is quadratic: four times the pixels, and a mask that is still guessing at the same place.
What actually helps is giving the edge a physical model.
What works on flat colour
Flat-colour art comes with a gift that photographs don't: the background is a known constant. You can sample it from the border, and once you have it, the compositing equation has structure:
C_observed = α · C_fg + (1 − α) · C_bg
Two unknowns per pixel (α and C_fg), three equations (R, G, B) — underdetermined in general, but not in practice, because the edge is spatially coherent. Alpha along an edge should vary smoothly in the direction tangential to the silhouette, and the foreground colour should be roughly constant along the normal.
That gives you the constraints a segmentation mask throws away:
- Unmix instead of delete. An edge pixel is not "mostly background, so remove it". It is a measured mixture, and it can be inverted — which is how thin outlines survive.
- Constrain alpha tangentially. Adjacent pixels along the silhouette should not disagree by 0.5. When they do, that disagreement is the fringe, and smoothing it removes the fringe without softening the edge.
- Re-solve colour after alpha. Once alpha is consistent, unmix with a clamp-aware solve rather than a raw division, so a slightly low alpha produces a slightly dark colour instead of a negative one.
None of this requires a model. It requires treating the edge as arithmetic.
Three checks you can run on any cutout
You don't need our code to evaluate a background remover. Three checks catch most of it:
- Cross the silhouette. Sample a horizontal line through a clean edge and print the alpha values. You want a ramp of four or more distinct values. Two values means thresholding.
- Put it on white, then on black. A correctly unmatted edge looks the same on both. A fringe that changes colour between the two backgrounds is background colour still baked into the pixels.
- Run it twice. Identical output both times means a deterministic pipeline. Different output means a model is resampling your asset, and your build artefacts are no longer reproducible.
The third one matters more than it sounds. If you are generating sprites for a game build, "the same input produced a slightly different sprite" is not a rounding error — it is an untracked change in your repository.
What this means in practice
Neural matting is genuinely good at photographs of people and products, where the subject is textured, the background is messy, and a plausible guess is better than a precise one.
Flat-colour art is the opposite case. The background is known, the colours are exact, and the edges are the whole point. Here, a model's willingness to guess is a liability, and the geometry of the problem is fully solvable.
That is the case our engine is built for: sample the background, invert the mixture, keep the shadow, keep the outline, and produce the same pixels every time.
If you want to see the difference on your own asset, drop it into the remover — it runs locally, so nothing is uploaded.