Repository navigation
Photos alias when shrunk and turn blocky under camera zoom: image and video sample with nearest neighbour and no mipmaps #465
Description
Activity
Working on this.
Mini-spec
Goal. One sampling decision, taken from the on-screen size the bitmap is finally drawn at, applied everywhere a raster bitmap is scaled to a destination rect.
Shape.
renderer::photo:photo_sampling(source, on_screen) -> SamplingOptions(pure, mipmapped linear when either axis minifies, Mitchell cubic otherwise) plusdraw_photo(canvas, image, dst, paint), which reads the on-screen size offcanvas.local_to_device_as_3x3()so the camera zoom is included. The axis scales come fromhypot(scale_x, skew_y)rather thanscale_xalone, so a rotated matrix does not report a shrink that is not there.Invariants.
- A bitmap drawn at or above source size never samples with mipmaps (cubic does not use them).
- The sampling never depends on anything but source size and on-screen size, so the same node under a camera and without one is decided on the same rule.
- No call site decides for itself.
Scope. 14 call sites:
image,video,avatar,avatar_group,badge,list,stat,mockup,gif,lottie,svgraster fallback,icon,transition::render_layer, and thesheetcommand.Out of scope, with the reason:
transition.rs:273andengine/render/scene.rs:207composite a full frame buffer at 1:1 through the point form ofdraw_image. Nothing is scaled, so the sampling is never consulted.paint_pass.rs:934draws a layer snapshot, also through the point form. It is under a scaling matrix, so it has the same defect, but a snapshot of already-rasterised content is a different question from a photo and the point form takes no dst rect. Left out deliberately.engine/text/cosmic.rs:196is the cosmic-text bridge, which CLAUDE.md records as not wired into real rendering.
Acceptance.
- A 640px checkerboard drawn at 64px (scale 1/10) reads mean luma 127 ± 2 with no near-black or near-white pixel, against 255 solid white measured on
main. - The same at 120px reads mean 127 ± 2, against mean 124 with min 0 / max 255 on
main. - A 640px source drawn into a 640px rect on a canvas scaled to 0.1 picks the minifying path: the camera zoom is read, not just the layout size.
- An enlargement picks cubic, not mipmaps.
Verify (exit).
cargo fmt --all --check;cargo clippy --workspace --all-targets --features rustmotion/studio -- -D warnings;cargo test --workspace --features rustmotion/studio.Risk. Mipmap generation costs memory and time per image on first draw. Measured before the PR goes up.
Fixed in #468.
Twenty call sites rather than the eleven listed:
video,lottieandsvgeach had more than one, andtransition::render_layeris reached throughsurface.canvas()so a grep forcanvas.draw_image_rectmisses it.iconwent through the helper too — it already asked for Mitchell, but unconditionally, so a shrunk icon had no mipmaps.Verified: the checkerboard reads mean 127 at both 64px and 120px where it read 255 solid and 0-to-255 noise before, measured through the
imagecomponent end to end and confirmed red before the fix. Two of the unit tests assert the old path's numbers so the fix cannot quietly become a no-op. A 640px rect on a canvas scaled to 0.1 confirms the camera zoom is read, not just the layout size. Full suite: 2153 passed, 0 failed.Measured cost, 300 frames at 1920x1080: twelve photos at 52px go 1.01s to 1.13s, one 640px source filling the frame goes 1.79s to 2.59s. The enlargement is where Mitchell is paid for.
Not done, and named in the PR:
paint_pass.rs:934draws a layer snapshot under a scaling matrix through the point form ofdraw_image, so it has the same defect. A snapshot of already-rasterised content is a different question from a photo and the point form takes no dst rect, so it is left out rather than guessed at. Say if you want an issue for it.
imageandvideodraw their bitmap withcanvas.draw_image_rect(img, None, dst, &Paint::default()), which samples with Skia's defaultSamplingOptions: nearest neighbour, no mipmaps. A photo drawn smaller than its source picks one source pixel out of N and drops the rest, so it aliases (jagged hair, stair-stepped glasses, moiré on fabric). A photo drawn larger, for instance under a camerazoom, is enlarged pixel by pixel into visible blocks.crates/rustmotion-components/src/image.rs:video.rs(draw_fitted) has the same call. The camera is applied as a canvas matrix (apply_camera_transformscales the canvas), so the scale the image is finally drawn at is the layout scale times the camera zoom, and both go through that nearest-neighbour sampling.Reproduction
A 640x640 checkerboard of 1px black/white cells (
checker.png), drawn at 64px and 120px on a#808080background. Correct downsampling averages it to the background grey.{ "version": "1.0", "video": { "width": 320, "height": 160, "fps": 30, "background": "#808080" }, "scenes": [{ "duration": 1.0, "layout": { "direction": "row", "align_items": "center", "justify_content": "center", "gap": 40 }, "children": [ { "type": "image", "src": "checker.png", "fit": "cover", "style": { "width": 64, "height": 64 } }, { "type": "image", "src": "checker.png", "fit": "cover", "style": { "width": 120, "height": 120 } } ] }] }Luma inside each image (4px inset),
render --frame 0:At 64px the scale is exactly 1/10, so every sampled pixel lands on the same parity and the image renders solid white. At 120px it is black/white noise.
Seen in production on a video using portraits: 640px faces drawn as 52px avatars in a table had visibly jagged hair and glasses, and a wall of 400px portraits under a
zoom: 3.4camera move showed pixel blocks.Proposed fix
Choose the sampling from the on-screen size, camera included:
SamplingOptions::new(FilterMode::Linear, MipmapMode::Linear);SamplingOptions::from(CubicResampler::mitchell())(cubic does not use mipmaps, so it is only right when enlarging).The on-screen width is
dst.width()times the scale ofcanvas.local_to_device_as_3x3(), which already includes the camera zoom. A shared helper inrustmotion-core::engine::renderer(draw_photo(canvas, image, dst, paint)) can replace thedraw_image_rectcall inimage.rsandvideo.rs. A local patch along those lines gives the "mipmapped" rows above.icon.rsalready does something similar withCubicResampler::mitchell().The other places that draw a bitmap with
Paint::default()are not covered by that patch and likely have the same problem:avatar.rs,avatar_group.rs,badge.rs,list.rs,stat.rs,mockup.rs,gif.rs,svg.rs(raster fallbacks),lottie.rs,engine/transition.rsand thesheetcommand.