From 79f3550f983f61b3973d4247bc1250c9163960ad Mon Sep 17 00:00:00 2001 From: tmathern <60901087+tmathern@users.noreply.github.com> Date: Tue, 3 Nov 2026 08:23:13 -0800 Subject: [PATCH 1/4] fix: Initial commit --- README.md | 10 +- impl-notes.md | 185 ++++++++++++++ rust/src/image_processing.rs | 454 ++++++++++++++++++++++++++++++----- rust/src/lib.rs | 39 ++- 4 files changed, 608 insertions(+), 80 deletions(-) create mode 100644 impl-notes.md diff --git a/README.md b/README.md index f48c081..1a154a5 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ pip install . ## Quickstart To get started quickly, run the `python/test-decode.py` script that provides examples of watermarking several -image files from the `images` directory. +image files from the `images` directory. ### Run the example @@ -105,15 +105,15 @@ im_recover.save('images/recovered.png') ## Data schema TrustMark encodes a payload (the watermark data embedded within the image) of 100 bits. -You can configure an error correction level over the raw 100 bits of payload to maintain reliability under transformations or noise. +You can configure an error correction level over the raw 100 bits of payload to maintain reliability under transformations or noise. -In payload encoding, the version bits comprise two reserved (unused) bits, and two bits encoding an integer value 0-3 that specifies the data schema as follows: +In payload encoding, the version bits comprise two reserved (unused) bits, and two bits encoding an integer value 0-3 that specifies the data schema as follows: - 0: BCH_SUPER - 1: BCH_5 - 2: BCH_4 - 3: BCH_3 -For more details and information on configuring the encoding mode in Python, see [Configuring TrustMark](python/CONFIG.md). +For more details and information on configuring the encoding mode in Python, see [Configuring TrustMark](python/CONFIG.md). ## Citation @@ -140,6 +140,6 @@ month = nov } ``` -## License +## License This package is is distributed under the terms of the [MIT license](https://github.com/adobe/trustmark/blob/main/LICENSE). The MIT license covers both the code in this repo and the model files which are downloaded upon first use. diff --git a/impl-notes.md b/impl-notes.md new file mode 100644 index 0000000..5f26159 --- /dev/null +++ b/impl-notes.md @@ -0,0 +1,185 @@ +# Lower peak memory in the Rust `Trustmark::encode` + +This note explains a change to the Rust crate in `rust/`: what was changed, the arithmetic the new +code performs, and where the memory saving comes from. It is written for engineers who know Rust +but not image processing. + +## What changed and what did not + +Two files changed: `rust/src/image_processing.rs` and `rust/src/lib.rs`. + +The public interface of the crate is unchanged. `Trustmark::encode` and `Trustmark::decode` keep +their signatures. `encode` still returns an `Rgb32F` image, or `Rgba32F` when the input has an +alpha channel, with the same color space metadata as before. Every value in the returned image is +bit-for-bit identical to what trustmark 0.4.0 returns for the same input, strength and model. + +The changes are internal: + +- `ModelImage` borrows the input image instead of owning a copy, which removes the two + `img.clone()` calls in `encode`. +- `encode` crops and resizes the input to the 256x256 model input once and reuses that array for + the residual; 0.4.0 did it twice for the same result. +- `apply_residual` is rewritten. It takes the residual as the array the model produced and writes + the result one row at a time, without building full-size floating point copies of the input or + of the resized residual. +- `encode` returns an error for an image with a width or height of 0, where 0.4.0 panicked. + +## How a watermark is added + +### Pixel values and the residual + +An image is a grid of pixels, and each pixel has a red, a green and a blue value (and sometimes an +alpha value for transparency, which the watermark never touches). The code works with these values +as numbers between 0 and 1: 0 is no light, 1 is full intensity. + +The watermarking model only accepts images of 256 by 256 pixels. `encode` shrinks the input to that +size, runs the model, and subtracts the model's input from its output. The difference is the +residual: a 256 by 256 grid of small corrections, one per channel, which encode the watermark. The +code multiplies it by the requested strength and limits each correction to plus or minus 0.2. For +the P variant, and for Q images more than twice as wide as tall (or the reverse), the residual is +padded to match the image's proportions. + +This is Algorithm 1 of the TrustMark paper (arXiv 2311.18297): compute the residual at the model's +size, enlarge it to the image's size, add it to the original image. + +### Enlarging the residual + +The residual has to be stretched to the size of the real image, for example from 256 to 6000 +columns. Each column of the large image falls somewhere between two columns of the small residual, +and its value is a blend of those two, weighted by distance. A position 30% of the way from column +4 to column 5 gets 70% of column 4's value and 30% of column 5's. Positions past the first or last +column take the edge value. Rows are handled the same way. This is bilinear interpolation, which +`image::imageops::resize` performs with `FilterType::Triangle`. + +When the image is smaller than the residual, the same function averages over a wider window of +residual columns, so every residual value contributes. + +`imageops::resize` does the two directions in sequence: it first resizes every column to the new +height, then every row to the new width. Because each output row only needs the two or so residual +rows nearest to it, the new code can produce the resized residual one output row at a time. + +### Adding it to the image + +For each pixel and each color channel, 0.4.0 computed, in 32-bit floating point: + +```text +x = 2 * pixel - 1 pixel value moved from [0, 1] to [-1, 1] +r = 2 * residual - 1 residual value moved from [0, 1] to [-1, 1] +out = (min(x + r, 1) + 1) / 2 +``` + +The residual is stored in the `[0, 1]` form, `(r + 1) / 2`, because 0.4.0 held it in an image +type. The new code keeps that form, and the same three lines, so it performs the same +floating point operations in the same order. + +### Why the arithmetic is repeated operation for operation + +The planning document for this change proposed a shorter formula that gives the same result in +exact arithmetic, accepting a difference of one 8-bit step in some pixels. That difference is too +large here. The figures in the paper show residuals scaled up 20 and 100 times to make them +visible, so the watermark changes a typical pixel by only a few 8-bit steps, and a rounding change +of one step is a noticeable part of the signal. + +Floating point results depend on the order of operations. `(a + b) + c` and `a + (b + c)` can +differ in the last bit, and after conversion to 8 bits that can move a pixel by one step. The new +code therefore computes the blending weights exactly as `imageops::resize` does in image 0.25.10 +(`vertical_sample` and `horizontal_sample`): + +- the same scale factor and center position; +- the same window of input positions, including those with zero weight; +- the same rescaling of the weights so they add up to 1; +- the same summation order; +- the same limit to `[0, 1]` after the horizontal pass; +- the same shortcut, a plain copy, when the residual already has the image's size. + +Input pixels are converted to floating point by the `image` crate itself, one row at a time, with +the same conversion 0.4.0 applied to the whole image. + +To check that the tests detect a change of this size, one run removed only the rescaling of the +weights, a change that affects nothing but rounding. The bit-for-bit test then reported 6132 of +1,048,576 values different on `ghost.png`. + +## Memory + +### Where 0.4.0 spent it + +For an input of N pixels, 0.4.0 allocated, among others: + +- two copies of the input image in `encode`, used only to shrink them to 256 by 256; +- in `apply_residual`, two 32-bit RGBA copies of the input (16 bytes per pixel each); +- the residual enlarged to full size as 32-bit RGB (12 bytes per pixel), then converted to RGBA + (16 bytes per pixel); +- at the end, a 32-bit RGB copy of the input and a 32-bit RGB copy of the result. + +Several of these are alive at the same moment, on top of the decoded input. + +### Where the new code spends it + +- The input image, as the caller decoded it, consumed by `encode` (3 bytes per pixel for an + ordinary RGB JPEG). +- The returned image (12 bytes per pixel for RGB, 16 with alpha), a size set by the public return + type. +- Per row: one row of the input converted to floating point, and one row of the residual resized + to the output height. +- The residual itself, 256 by at most a few hundred values. + +### Measured + +Measured with `/usr/bin/time -l` on macOS, on a 6000 by 4000 RGB JPEG (24 million pixels), model Q, +two runs each. "Baseline" loads the models and decodes the image without encoding. + +| Run | Peak resident memory | Above baseline, per pixel | +| --- | --- | --- | +| Baseline | 223 MB | | +| 0.4.0 `encode` | 1868 MB | 68.5 bytes | +| new `encode` | 589 MB | 15.3 bytes | + +The 15.3 bytes include the 12-byte output and the inference workspace of the encoder model, which +these runs do not separate. Run time for the same image went from 0.57 to 0.48 seconds. The crate's +`cargo bench` (`benches/encode.rs`, a 240 by 240 image) measured 96.7 ms for 0.4.0 and 90.6 ms for +the new code. + +Going lower requires an 8-bit result, and `encode` returns 32-bit floating point values. An +additional method that returns `Rgb8` or `Rgba8`, written into the input buffer, would bring the +cost to about 3 bytes per pixel above the decoded image. It is not part of this change. + +## How it was checked + +- Unit tests in `image_processing.rs` compare `apply_residual` with the 0.4.0 implementation, kept + in the test module as `apply_residual_reference`, and require identical bytes, color type, + dimensions and color space. They cover ten pixel formats (8-bit, 16-bit and 32-bit, gray and + color, with and without alpha), image sizes larger than, equal to, smaller than and mixed against + the residual, three residual shapes, values outside `[0, 1]`, and residuals outside the range + `encode` produces. +- An end-to-end comparison built 0.4.0 and the new code into one program and ran `encode` with the + real Q and P models on 28 inputs: the sample images, gray, 16-bit and floating point versions of + `ripley.jpg`, and synthetic images from 1x1 to 4000x10. All outputs were identical, as were the + decode results. Two runs of 0.4.0 agreed with each other, so the comparison is not hiding + nondeterminism. +- The same program compared the two versions on NaN, infinite, negative, zero and large strengths, + on images exactly at and one pixel past the 2:1 aspect limit, and on images with Display P3 + primaries and a linear transfer function. All outputs were identical. +- The Python implementation in `python/` decoded all 20 Rust-encoded images (Q and P, from 0.4.0 + and from the new code), and both Rust versions decoded the 6 images encoded by Python. + Unwatermarked images decoded as unwatermarked in both languages. The Python encoder uses a + different algorithm (it removes the per-channel mean of the residual and uses PyTorch's + interpolation), so its output bytes differ from Rust's, as they did with 0.4.0. Image quality is + similar: 41.7 to 44.2 dB PSNR for Rust and 41.8 to 44.6 dB for Python with model Q, 46.8 to + 50.5 dB and 47.0 to 50.8 dB with model P. +- `cargo clippy --workspace --all-targets -- -D warnings` and `cargo fmt --check` report nothing, + and `cargo test --workspace` passes. + +## Behavior to know about + +One case changed. `encode` on an image with a width or height of 0 panicked inside `ndarray` in +0.4.0. The new code returns `Error::ImageProcessing(image_processing::Error::Image)` ("invalid image") +before running the model. `decode` on such an image returns `Error::CorruptWatermark`, in 0.4.0 +and in the new code. + +The following cases behave the same in both versions; they are listed because a reader may not +expect them. + +- An image of 100 by 60 pixels encodes, but its watermark does not decode, with either version. +- A residual with an empty dimension makes the internal `apply_residual` return + `Error::InvalidShape`. 0.4.0 resized such a residual to zeros instead. `encode` cannot produce + one: the model returns 256 by 256 values, and the padding only adds to that. diff --git a/rust/src/image_processing.rs b/rust/src/image_processing.rs index 69a7a61..33b51a3 100644 --- a/rust/src/image_processing.rs +++ b/rust/src/image_processing.rs @@ -9,11 +9,10 @@ use std::cmp; use fast_image_resize::{ResizeAlg, ResizeOptions, Resizer}; use image::{ - imageops::{self, FilterType}, - DynamicImage, GenericImageView as _, GrayAlphaImage, GrayImage, ImageBuffer, Pixel as _, - Rgb32FImage, RgbImage, Rgba32FImage, RgbaImage, + DynamicImage, GenericImageView as _, GrayAlphaImage, GrayImage, ImageBuffer, Rgb32FImage, + RgbImage, Rgba32FImage, RgbaImage, }; -use ndarray::{s, Array, ArrayD, Axis, ShapeError}; +use ndarray::{s, Array, Array2, Array3, Array4, ArrayD, Axis, ShapeError}; use ort::value::{TensorValueType, Value}; use crate::Variant; @@ -34,7 +33,11 @@ macro_rules! convert_from_neg1_1_to_0_1 { }; } -pub(super) struct ModelImage(pub(super) u32, pub(super) Variant, pub(super) DynamicImage); +pub(super) struct ModelImage<'a>( + pub(super) u32, + pub(super) Variant, + pub(super) &'a DynamicImage, +); /// The error type for the `image_processing` module. #[derive(Debug, thiserror::Error)] @@ -60,18 +63,27 @@ pub enum Error { Resize(#[from] fast_image_resize::ResizeError), } -impl TryFrom for Value> { +impl TryFrom> for Value> { type Error = Error; - fn try_from(ModelImage(size, variant, img): ModelImage) -> Result { - let (w, h, xpos, ypos) = center_crop_size_and_offset(variant, &img); + fn try_from(img: ModelImage<'_>) -> Result { + let array: Array4 = img.try_into()?; + Ok(Value::from_array(array)?) + } +} + +impl TryFrom> for Array4 { + type Error = Error; + + fn try_from(ModelImage(size, variant, img): ModelImage<'_>) -> Result { + let (w, h, xpos, ypos) = center_crop_size_and_offset(variant, img); let options = ResizeOptions::new() .crop(xpos as f64, ypos as f64, w as f64, h as f64) .resize_alg(ResizeAlg::Interpolation( fast_image_resize::FilterType::Bilinear, )); - let modified_img = resize_img(&img, size, size, options)?; + let modified_img = resize_img(img, size, size, options)?; let img = modified_img.into_rgb32f().into_vec(); let array = Array::from(img); @@ -85,74 +97,197 @@ impl TryFrom for Value> { .reversed_axes(); array.swap_axes(2, 3); assert_eq!(array.shape(), &[1, 3, size as usize, size as usize]); - Ok(Value::from_array(array.to_owned())?) + Ok(array.to_owned()) } } -impl TryFrom<(u32, Variant, ArrayD)> for ModelImage { - type Error = Error; - - fn try_from( - (size, variant, mut array): (u32, Variant, ArrayD), - ) -> Result { - let &[1, 3, height, width] = &array.shape().to_owned()[..] else { - return Err(Error::InvalidShape); - }; - array.swap_axes(2, 3); - let array = array.reversed_axes().remove_axis(Axis(3)); - let array = array.to_shape([width * height * 3])?; +/// Red, green and blue: the channels the residual has and is added to. Alpha gets no residual. +const COLOR_CHANNELS: usize = 3; - // The `image` crate normalizes to `[0,1]`. Trustmark wants images normalized to `[-1,1]`. - let array = convert_from_neg1_1_to_0_1!(array); +/// Apply `residual` to the `input`. +/// +/// `residual` has shape `[1, 3, height, width]` with values in `[-1, 1]`. It is resized to the +/// size of `input` and added to the color channels; alpha is kept. +/// +/// The output equals that of the 0.4.0 implementation (kept as `apply_residual_reference`) bit +/// for bit. The watermark signal is only a few 8-bit steps, so this repeats the arithmetic of +/// `image::imageops::resize` with `FilterType::Triangle` operation for operation, instead of an +/// equivalent formula that rounds differently. Unlike 0.4.0, it makes no full-size copies besides +/// the output: the residual is resized and the input converted one row at a time. +pub(super) fn apply_residual( + input: DynamicImage, + residual: &ArrayD, +) -> Result { + let &[1, COLOR_CHANNELS, residual_height, residual_width] = residual.shape() else { + return Err(Error::InvalidShape); + }; + // The model never produces an empty residual, and an empty one cannot be resized. + if residual_height == 0 || residual_width == 0 { + return Err(Error::InvalidShape); + } + // The residual as the 0.4.0 `Rgb32F` image held it: values in `[0, 1]`, indexed + // `[channel, row, column]`. + let residual = Array3::from_shape_fn( + (COLOR_CHANNELS, residual_height, residual_width), + |(channel, row, column)| convert_from_neg1_1_to_0_1!(residual[[0, channel, row, column]]), + ); + + let (width, height) = input.dimensions(); + // `imageops::resize` returns an unchanged copy when the size already matches. + let resize = + ((width as usize, height as usize) != (residual_width, residual_height)).then(|| { + ResizeWeights { + columns: bilinear_resize_weights(residual_width, width), + rows: bilinear_resize_weights(residual_height, height), + } + }); + + // The conversion gives the output the input's color space, as in 0.4.0; the color values + // are overwritten. + Ok(if input.color().has_alpha() { + let mut output = input.to_rgba32f(); + add_residual( + &mut output, + COLOR_CHANNELS + 1, + &input, + &residual, + resize.as_ref(), + ); + output.into() + } else { + let mut output = input.to_rgb32f(); + add_residual( + &mut output, + COLOR_CHANNELS, + &input, + &residual, + resize.as_ref(), + ); + output.into() + }) +} - let image = Rgb32FImage::from_vec(width as u32, height as u32, array.to_vec()) - .ok_or(Error::Image)?; +/// How to resize the residual to the image: one entry per output column and per output row. +struct ResizeWeights { + columns: Vec, + rows: Vec, +} - Ok(Self(size, variant, image.into())) - } +/// How one output position of a resize along one axis is computed: the weighted sum of the +/// input values at `first_input`, `first_input + 1`, and so on, one per weight. +struct WeightedInputs { + first_input: usize, + weights: Vec, } -/// Apply `residual` to the `input`. -/// -/// This function upscales `residual` to be the size of of `input`, then adds `residual` to the -/// `input`. -pub(super) fn apply_residual(input: DynamicImage, residual: DynamicImage) -> DynamicImage { - let has_alpha = input.color().has_alpha(); - let (w, h) = input.dimensions(); - - let applied = { - let input = input.clone().into_rgba32f(); - let mut target = input.clone(); - - let residual = residual.resize_exact(w, h, FilterType::Triangle); - let residual = residual.into_rgba32f(); - - for ((target, residual), original) in target - .pixels_mut() - .zip(residual.pixels()) - .zip(input.pixels()) - { - target.apply2(residual, |x, y| { - let x = convert_from_0_1_to_neg1_1!(x); - let y = convert_from_0_1_to_neg1_1!(y); - - convert_from_neg1_1_to_0_1!(f32::min(x + y, 1.0)) - }); - target[3] = original[3]; +impl WeightedInputs { + /// The weighted sum of the values `input` returns for each position. The additions run in + /// position order, as in `imageops::resize`, since the order changes the rounding. + fn sum(&self, input: impl Fn(usize) -> f32) -> f32 { + let mut sum = 0.0; + for (offset, weight) in self.weights.iter().enumerate() { + sum += input(self.first_input + offset) * weight; } + sum + } +} - target - }; +/// Resizes `input_size` positions to `output_size` positions by blending neighboring values, as +/// `imageops::resize` does with `FilterType::Triangle` (image 0.25, `vertical_sample` and +/// `horizontal_sample`). Computed the same way, so the results round the same way. +fn bilinear_resize_weights(input_size: usize, output_size: u32) -> Vec { + // How many input positions one output position covers. + let scale = input_size as f32 / output_size as f32; + // How far from its center an output position blends inputs: one position when enlarging, + // more when shrinking so that every input counts. + let reach = if scale < 1.0 { 1.0 } else { scale }; + + (0..output_size) + .map(|output_position| { + // Where the center of this output position falls, in input positions. + let center = (output_position as f32 + 0.5) * scale; + let first = ((center - reach).floor() as i64).clamp(0, input_size as i64 - 1); + let end = ((center + reach).ceil() as i64).clamp(first + 1, input_size as i64); + let center = center - 0.5; + + // Nearer inputs count more. Inputs past the edge are left out, and the weights + // rescaled to add up to 1. + let mut weights: Vec = (first..end) + .map(|input_position| triangle((input_position as f32 - center) / reach)) + .collect(); + let total = weights.iter().fold(0.0, |total, weight| total + weight); + for weight in &mut weights { + *weight /= total; + } + WeightedInputs { + first_input: first as usize, + weights, + } + }) + .collect() +} - if has_alpha { - let mut input = input.into_rgba32f(); - imageops::replace(&mut input, &applied, 0, 0); - input.into() +/// 1 at distance 0, falling in a straight line to 0 at distance 1 and beyond. +fn triangle(distance: f32) -> f32 { + if distance.abs() < 1.0 { + 1.0 - distance.abs() } else { - let mut input = input.into_rgb32f(); - let applied = DynamicImage::ImageRgba32F(applied).into_rgb32f(); - imageops::replace(&mut input, &applied, 0, 0); - input.into() + 0.0 + } +} + +/// Writes `input` plus the `residual` into the color channels of `output`, one row at a time. +/// `output` holds `channels` values per pixel. The residual is resized with `resize`, or used +/// as is when that is `None`. +fn add_residual( + output: &mut [f32], + channels: usize, + input: &DynamicImage, + residual: &Array3, + resize: Option<&ResizeWeights>, +) { + let width = input.width(); + let residual_width = residual.dim().2; + // One row of the residual, resized to the output height but not yet to the output width. + let mut residual_row = Array2::zeros((COLOR_CHANNELS, residual_width)); + + for (row, output_row) in output + .chunks_exact_mut(width as usize * channels) + .enumerate() + { + // Resize vertically first, as `imageops::resize` does (`vertical_sample`). + if let Some(resize) = resize { + for channel in 0..COLOR_CHANNELS { + for column in 0..residual_width { + residual_row[[channel, column]] = + resize.rows[row].sum(|source_row| residual[[channel, source_row, column]]); + } + } + } + + // 0.4.0 read the input converted to `Rgba32F`, whatever the output type. + let input_row = input.crop_imm(0, row as u32, width, 1).into_rgba32f(); + + for (column, output_pixel) in output_row.chunks_exact_mut(channels).enumerate() { + let input_pixel = input_row.get_pixel(column as u32, 0); + for channel in 0..COLOR_CHANNELS { + let residual_value = match resize { + // Then horizontally (`horizontal_sample`), which limits the result to + // `[0, 1]`. + Some(resize) => resize.columns[column] + .sum(|source_column| residual_row[[channel, source_column]]) + .clamp(0.0, 1.0), + None => residual[[channel, row, column]], + }; + + // The 0.4.0 `apply_residual` closure: both values to `[-1, 1]`, added, capped + // at 1, and back to `[0, 1]`. + let pixel_value = convert_from_0_1_to_neg1_1!(input_pixel[channel]); + let residual_value = convert_from_0_1_to_neg1_1!(residual_value); + output_pixel[channel] = + convert_from_neg1_1_to_0_1!(f32::min(pixel_value + residual_value, 1.0)); + } + } } } @@ -270,10 +405,73 @@ pub(super) fn remove_boundary_artifact( #[cfg(test)] mod tests { - use ndarray::Array4; + use image::{ + imageops::{self, FilterType}, + Pixel as _, + }; use super::*; + /// The 0.4.0 conversion of the residual array into an `Rgb32F` image, kept as the reference + /// for `apply_residual`. + fn residual_image_reference(mut array: ArrayD) -> Result { + let &[1, 3, height, width] = &array.shape().to_owned()[..] else { + return Err(Error::InvalidShape); + }; + array.swap_axes(2, 3); + let array = array.reversed_axes().remove_axis(Axis(3)); + let array = array.to_shape([width * height * 3])?; + + // The `image` crate normalizes to `[0,1]`. Trustmark wants images normalized to `[-1,1]`. + let array = convert_from_neg1_1_to_0_1!(array); + + let image = Rgb32FImage::from_vec(width as u32, height as u32, array.to_vec()) + .ok_or(Error::Image)?; + + Ok(image.into()) + } + + /// The 0.4.0 `apply_residual`, kept as the reference for `apply_residual`. + fn apply_residual_reference(input: DynamicImage, residual: DynamicImage) -> DynamicImage { + let has_alpha = input.color().has_alpha(); + let (w, h) = input.dimensions(); + + let applied = { + let input = input.clone().into_rgba32f(); + let mut target = input.clone(); + + let residual = residual.resize_exact(w, h, FilterType::Triangle); + let residual = residual.into_rgba32f(); + + for ((target, residual), original) in target + .pixels_mut() + .zip(residual.pixels()) + .zip(input.pixels()) + { + target.apply2(residual, |x, y| { + let x = convert_from_0_1_to_neg1_1!(x); + let y = convert_from_0_1_to_neg1_1!(y); + + convert_from_neg1_1_to_0_1!(f32::min(x + y, 1.0)) + }); + target[3] = original[3]; + } + + target + }; + + if has_alpha { + let mut input = input.into_rgba32f(); + imageops::replace(&mut input, &applied, 0, 0); + input.into() + } else { + let mut input = input.into_rgb32f(); + let applied = DynamicImage::ImageRgba32F(applied).into_rgb32f(); + imageops::replace(&mut input, &applied, 0, 0); + input.into() + } + } + #[test] fn renormalize_from_0_1() { assert_eq!(convert_from_0_1_to_neg1_1!(0.), -1.); @@ -334,4 +532,128 @@ mod tests { assert_eq!(output.shape(), &[1, 3, 298, 256]); } + + /// Deterministic values in `[0, 1)` (xorshift), so the tests need no random number crate. + fn noise(mut state: u32) -> impl FnMut() -> f32 { + move || { + state ^= state << 13; + state ^= state >> 17; + state ^= state << 5; + (state >> 8) as f32 / (1 << 24) as f32 + } + } + + fn residual(height: usize, width: usize, seed: u32) -> ArrayD { + let mut noise = noise(seed); + Array4::from_shape_simple_fn([1, 3, height, width], || noise() * 0.4 - 0.2).into_dyn() + } + + fn noise_image(width: u32, height: u32, seed: u32) -> Rgba32FImage { + let mut noise = noise(seed); + Rgba32FImage::from_fn(width, height, |_, _| { + image::Rgba([noise(), noise(), noise(), noise()]) + }) + } + + /// `apply_residual` output against the 0.4.0 implementation, bit for bit. + fn assert_matches_reference(input: &DynamicImage, residual: &ArrayD) { + let actual = apply_residual(input.clone(), residual).unwrap(); + let expected = apply_residual_reference( + input.clone(), + residual_image_reference(residual.clone()).unwrap(), + ); + assert_eq!(actual.color(), expected.color()); + assert_eq!(actual.dimensions(), expected.dimensions()); + assert_eq!(actual.color_space(), expected.color_space()); + // Both are `f32` images; compare each value's bytes. + let (actual, expected) = ( + actual.as_bytes().as_chunks::<4>().0, + expected.as_bytes().as_chunks::<4>().0, + ); + let mismatches = actual.iter().zip(expected).filter(|(a, e)| a != e).count(); + assert_eq!( + mismatches, + 0, + "{:?} {}x{}, residual {:?}: {mismatches} of {} values differ", + input.color(), + input.width(), + input.height(), + residual.shape(), + expected.len() + ); + } + + const RESIDUAL_SHAPES: [(usize, usize); 3] = [(256, 256), (256, 384), (384, 256)]; + + #[test] + fn apply_residual_matches_reference_on_sample_images() { + for path in ["../images/ghost.png", "../images/ufo_240.jpg"] { + let img = image::open(path).unwrap(); + for (seed, (h, w)) in RESIDUAL_SHAPES.into_iter().enumerate() { + assert_matches_reference(&img, &residual(h, w, seed as u32 + 1)); + } + } + } + + #[test] + fn apply_residual_matches_reference_on_every_color_type() { + // Larger than, equal to, smaller than and mixed against the residual sizes. + let sizes = [(600, 450), (256, 256), (256, 300), (100, 60), (200, 500)]; + for (i, (w, h)) in sizes.into_iter().enumerate() { + let float = DynamicImage::ImageRgba32F(noise_image(w, h, 100 + i as u32)); + let inputs = [ + DynamicImage::ImageLuma8(float.to_luma8()), + DynamicImage::ImageLumaA8(float.to_luma_alpha8()), + DynamicImage::ImageRgb8(float.to_rgb8()), + DynamicImage::ImageRgba8(float.to_rgba8()), + DynamicImage::ImageLuma16(float.to_luma16()), + DynamicImage::ImageLumaA16(float.to_luma_alpha16()), + DynamicImage::ImageRgb16(float.to_rgb16()), + DynamicImage::ImageRgba16(float.to_rgba16()), + DynamicImage::ImageRgb32F(float.to_rgb32f()), + float, + ]; + for input in &inputs { + for (seed, (rh, rw)) in RESIDUAL_SHAPES.into_iter().enumerate() { + assert_matches_reference(input, &residual(rh, rw, seed as u32 + 10)); + } + } + } + } + + #[test] + fn apply_residual_matches_reference_on_out_of_range_floats() { + let mut noise = noise(7); + let img = Rgb32FImage::from_fn(300, 280, |_, _| { + image::Rgb([ + noise() * 3.0 - 1.0, + noise() * 3.0 - 1.0, + noise() * 3.0 - 1.0, + ]) + }); + assert_matches_reference(&DynamicImage::ImageRgb32F(img), &residual(256, 256, 3)); + } + + #[test] + fn apply_residual_matches_reference_on_out_of_range_residual() { + // `encode` limits the residual to `[-0.2, 0.2]`; `apply_residual` must still match 0.4.0 + // beyond that, with and without resizing. + let residual = residual(256, 256, 4).mapv(|value| value * 8.0); + for (w, h) in [(256, 256), (300, 280)] { + let img = DynamicImage::ImageRgba32F(noise_image(w, h, 9)); + assert_matches_reference(&img, &residual); + } + } + + #[test] + fn apply_residual_rejects_bad_shape() { + for shape in [[1, 4, 256, 256], [1, 3, 0, 256], [1, 3, 256, 0]] { + let img = DynamicImage::ImageRgb8(RgbImage::new(10, 10)); + let residual = Array4::::zeros(shape).into_dyn(); + assert!(matches!( + apply_residual(img, &residual), + Err(Error::InvalidShape) + )); + } + } } diff --git a/rust/src/lib.rs b/rust/src/lib.rs index ce66dcb..63dde74 100644 --- a/rust/src/lib.rs +++ b/rust/src/lib.rs @@ -119,6 +119,8 @@ impl Trustmark { /// `watermark` is a bitstring encoding the watermark identifier to encode. `img` is the image /// which will be watermarked. `strength` is a number between 0 and 1 indicating how strong the /// resulting watermark should be. 0.95 is a normal strength. + /// + /// An image with a width or height of 0 is an error. pub fn encode( &self, watermark: String, @@ -126,13 +128,21 @@ impl Trustmark { strength: f32, ) -> Result { let (original_width, original_height) = img.dimensions(); + if original_width == 0 || original_height == 0 { + return Err(image_processing::Error::Image.into()); + } let aspect_ratio = original_width as f32 / original_height as f32; // the image is always encoded with size 256x256 let encode_size = 256; + // The model input is also needed for the residual; cloning it is cheaper than redoing the + // crop and resize. + let model_input: ndarray::Array4 = + ModelImage(encode_size, self.variant, &img).try_into()?; + // Same error variant as when the `ModelImage` conversion created the tensor. let input_img: Value> = - ModelImage(encode_size, self.variant, img.clone()).try_into()?; + Value::from_array(model_input.clone()).map_err(image_processing::Error::from)?; let bits: Value> = Bits::apply_error_correction_and_schema(watermark, self.version)?.into(); let output_img = { @@ -148,10 +158,8 @@ impl Trustmark { }; // Need to calculate and apply the residual. - let input_img: Value> = - ModelImage(encode_size, self.variant, img.clone()).try_into()?; - let residual = (self.variant.strength_multiplier() * strength) - * (output_img - input_img.try_extract_array::()?); + let residual = + (self.variant.strength_multiplier() * strength) * (output_img - model_input.into_dyn()); // Residual should be small perturbations. let mut residual = residual.clamp(-0.2, 0.2); @@ -165,9 +173,7 @@ impl Trustmark { ); } - let ModelImage(_, _, residual) = (encode_size, self.variant, residual).try_into()?; - - Ok(image_processing::apply_residual(img, residual)) + Ok(image_processing::apply_residual(img, &residual)?) } /// Decode a watermark from an image. @@ -176,7 +182,7 @@ impl Trustmark { let decode_size = if self.variant == Variant::P { 224 } else { 256 }; let img: Value> = - ModelImage(decode_size, self.variant, img).try_into()?; + ModelImage(decode_size, self.variant, &img).try_into()?; let watermark = { let mut decoder = self .decoder @@ -215,6 +221,21 @@ mod tests { assert_eq!(watermark, decoded); } + #[test] + fn encode_rejects_empty_image() { + for variant in [Variant::Q, Variant::P] { + let tm = Trustmark::new("./models", variant, Version::Bch5).unwrap(); + for (width, height) in [(0, 0), (0, 10), (10, 0)] { + let input = DynamicImage::new_rgb8(width, height); + let result = tm.encode("0010101".to_owned(), input, 0.95); + assert!(matches!( + result, + Err(Error::ImageProcessing(image_processing::Error::Image)) + )); + } + } + } + #[test] fn roundtrip_ghost() { roundtrip("../images/ghost.png"); From 691e7811a5b7c07d9ca3a541874659c54b717bda Mon Sep 17 00:00:00 2001 From: tmathern <60901087+tmathern@users.noreply.github.com> Date: Wed, 4 Nov 2026 18:57:19 -0800 Subject: [PATCH 2/4] fix: Implementation notes --- impl-notes.md | 271 ++++++++++++-------- rust/src/image_processing.rs | 463 ++++++++--------------------------- rust/src/lib.rs | 37 ++- 3 files changed, 292 insertions(+), 479 deletions(-) diff --git a/impl-notes.md b/impl-notes.md index 5f26159..9f1b3b0 100644 --- a/impl-notes.md +++ b/impl-notes.md @@ -1,17 +1,18 @@ # Lower peak memory in the Rust `Trustmark::encode` -This note explains a change to the Rust crate in `rust/`: what was changed, the arithmetic the new -code performs, and where the memory saving comes from. It is written for engineers who know Rust -but not image processing. +This note explains a change to the Rust crate in `rust/`: what was changed, the arithmetic the code +performs, and where the memory saving comes from. It is written for engineers who know Rust but +not image processing. ## What changed and what did not Two files changed: `rust/src/image_processing.rs` and `rust/src/lib.rs`. -The public interface of the crate is unchanged. `Trustmark::encode` and `Trustmark::decode` keep -their signatures. `encode` still returns an `Rgb32F` image, or `Rgba32F` when the input has an -alpha channel, with the same color space metadata as before. Every value in the returned image is -bit-for-bit identical to what trustmark 0.4.0 returns for the same input, strength and model. +The public interface of the crate is unchanged. `Trustmark::new`, `Trustmark::encode` and +`Trustmark::decode` keep their signatures, and the error enums keep their variants. `encode` +still returns an `Rgb32F` image, or `Rgba32F` when the input has an alpha channel, with the same +color space metadata as before. Every value in the returned image is bit-for-bit identical to what +trustmark 0.4.0 returns for the same input, strength and model. The changes are internal: @@ -19,10 +20,14 @@ The changes are internal: `img.clone()` calls in `encode`. - `encode` crops and resizes the input to the 256x256 model input once and reuses that array for the residual; 0.4.0 did it twice for the same result. -- `apply_residual` is rewritten. It takes the residual as the array the model produced and writes - the result one row at a time, without building full-size floating point copies of the input or - of the resized residual. -- `encode` returns an error for an image with a width or height of 0, where 0.4.0 panicked. +- `remove_boundary_artifact` stores the padded residual in pixel order, and `residual_image` + converts it into an image in place. 0.4.0 held three copies of the padded residual at once. +- `apply_residual` writes the result into the input converted to `Rgb32F` (or `Rgba32F` with + alpha). 0.4.0 made two `Rgba32F` copies of the input, converted the resized residual to + `Rgba32F`, and copied the result into a third buffer at the end. The residual is still resized + by the same `image` call as in 0.4.0. +- With the P and Q variants, `encode` returns an error for an image with a width or height of 0, + where 0.4.0 panicked. ## How a watermark is added @@ -35,13 +40,27 @@ as numbers between 0 and 1: 0 is no light, 1 is full intensity. The watermarking model only accepts images of 256 by 256 pixels. `encode` shrinks the input to that size, runs the model, and subtracts the model's input from its output. The difference is the residual: a 256 by 256 grid of small corrections, one per channel, which encode the watermark. The -code multiplies it by the requested strength and limits each correction to plus or minus 0.2. For -the P variant, and for Q images more than twice as wide as tall (or the reverse), the residual is -padded to match the image's proportions. +code multiplies it by the requested strength and limits each correction to plus or minus 0.2. This is Algorithm 1 of the TrustMark paper (arXiv 2311.18297): compute the residual at the model's size, enlarge it to the image's size, add it to the original image. +### Padding the residual + +For the P variant, and for Q images more than twice as wide as tall (or the reverse), the model +only saw a square cut from the middle of the image. `remove_boundary_artifact` then builds a +larger residual with the proportions of the whole image: the 256 by 256 residual sits in the +middle, and the rest is filled with the residual's average value for each channel. For a 1000 by +250 image this padded residual is 1024 by 256. + +The padded residual is an array with the shape `[1, 3, height, width]`: all red values first, then +all green, then all blue. An image buffer stores the three values of each pixel next to each +other. 0.4.0 allocated the array in the first order and copied it twice while converting it into +an image. The array now keeps the same shape and the same values, but its memory is laid out in +pixel order, so `residual_image` converts each value from `[-1, 1]` to `[0, 1]` in place and hands +the buffer to the image without copying it. Arrays in the other order, such as the unpadded 256 by +256 residual, are copied once. + ### Enlarging the residual The residual has to be stretched to the size of the real image, for example from 256 to 6000 @@ -49,14 +68,8 @@ columns. Each column of the large image falls somewhere between two columns of t and its value is a blend of those two, weighted by distance. A position 30% of the way from column 4 to column 5 gets 70% of column 4's value and 30% of column 5's. Positions past the first or last column take the edge value. Rows are handled the same way. This is bilinear interpolation, which -`image::imageops::resize` performs with `FilterType::Triangle`. - -When the image is smaller than the residual, the same function averages over a wider window of -residual columns, so every residual value contributes. - -`imageops::resize` does the two directions in sequence: it first resizes every column to the new -height, then every row to the new width. Because each output row only needs the two or so residual -rows nearest to it, the new code can produce the resized residual one output row at a time. +`image::imageops::resize` performs with `FilterType::Triangle`. Both 0.4.0 and the new code call it +through `DynamicImage::resize_exact`. ### Adding it to the image @@ -68,36 +81,34 @@ r = 2 * residual - 1 residual value moved from [0, 1] to [-1, 1] out = (min(x + r, 1) + 1) / 2 ``` -The residual is stored in the `[0, 1]` form, `(r + 1) / 2`, because 0.4.0 held it in an image -type. The new code keeps that form, and the same three lines, so it performs the same -floating point operations in the same order. - -### Why the arithmetic is repeated operation for operation - -The planning document for this change proposed a shorter formula that gives the same result in -exact arithmetic, accepting a difference of one 8-bit step in some pixels. That difference is too -large here. The figures in the paper show residuals scaled up 20 and 100 times to make them -visible, so the watermark changes a typical pixel by only a few 8-bit steps, and a rounding change -of one step is a noticeable part of the signal. - -Floating point results depend on the order of operations. `(a + b) + c` and `a + (b + c)` can -differ in the last bit, and after conversion to 8 bits that can move a pixel by one step. The new -code therefore computes the blending weights exactly as `imageops::resize` does in image 0.25.10 -(`vertical_sample` and `horizontal_sample`): - -- the same scale factor and center position; -- the same window of input positions, including those with zero weight; -- the same rescaling of the weights so they add up to 1; -- the same summation order; -- the same limit to `[0, 1]` after the horizontal pass; -- the same shortcut, a plain copy, when the residual already has the image's size. - -Input pixels are converted to floating point by the `image` crate itself, one row at a time, with -the same conversion 0.4.0 applied to the whole image. - -To check that the tests detect a change of this size, one run removed only the rescaling of the -weights, a change that affects nothing but rounding. The bit-for-bit test then reported 6132 of -1,048,576 values different on `ghost.png`. +The new code runs the same three lines, in the same order, on each pixel of the converted input, +and overwrites the color values in place. Alpha values are not written. 0.4.0 also left alpha +unchanged: it computed the formula on all four channels and then copied the original alpha back. + +### Why the output is bit-for-bit identical + +The watermark changes a typical pixel by only a few 8-bit steps; the figures in the paper show +residuals scaled up 20 and 100 times to make them visible. Floating point results depend on the +order of operations, and after conversion to 8 bits a rounding difference can move a pixel by one +step, a noticeable part of that signal. So the change keeps every floating point operation of +0.4.0: + +- The padded residual holds the same values; only the order of the values in memory changed. +- The conversion of the residual to `[0, 1]` is the same `(value + 1) / 2` on each value. A probe + ran all 4,294,967,296 possible 32-bit floating point values, NaN and infinities included, + through the 0.4.0 conversion and the new one, in both memory orders, and found no differing bit. +- The residual is resized by the same `image` call. +- The input is converted by the `image` crate. With alpha, this is the same `Rgba32F` conversion + 0.4.0 used. Without alpha, the new code converts to `Rgb32F`, where 0.4.0 read from an + `Rgba32F` copy. The two conversions give the same color values: a probe compared them bit for + bit for every 8-bit and 16-bit value in all ten pixel formats, and for 4,006 floating point + values including NaN and infinities, in sRGB, Display P3 and linear color spaces, and found no + difference. +- The sum is the 0.4.0 formula. + +An earlier version of this change also replaced `imageops::resize` with code that produced the +resized residual one row at a time. It saved about 13 more bytes per pixel and was bit-identical, +but it reproduced the internals of the `image` crate. It was dropped in favor of calling the crate. ## Memory @@ -106,6 +117,7 @@ weights, a change that affects nothing but rounding. The bit-for-bit test then r For an input of N pixels, 0.4.0 allocated, among others: - two copies of the input image in `encode`, used only to shrink them to 256 by 256; +- three copies of the padded residual while converting it into an image; - in `apply_residual`, two 32-bit RGBA copies of the input (16 bytes per pixel each); - the residual enlarged to full size as 32-bit RGB (12 bytes per pixel), then converted to RGBA (16 bytes per pixel); @@ -116,70 +128,123 @@ Several of these are alive at the same moment, on top of the decoded input. ### Where the new code spends it - The input image, as the caller decoded it, consumed by `encode` (3 bytes per pixel for an - ordinary RGB JPEG). + ordinary RGB JPEG). It is freed once converted. +- One copy of the padded residual, freed once resized. +- The residual enlarged to full size as 32-bit RGB (12 bytes per pixel). - The returned image (12 bytes per pixel for RGB, 16 with alpha), a size set by the public return type. -- Per row: one row of the input converted to floating point, and one row of the residual resized - to the output height. -- The residual itself, 256 by at most a few hundred values. +- The intermediate buffer of the resize, 16 bytes per value for the residual's width times the + image's height. + +The padded residual has 256 by (256 times the long side divided by the short side) values per +channel, about 786 KB for each unit of that ratio. For most photographs it is a few megabytes. +For very thin images it is larger than the image itself: a 1 by 2000 image gets a 256 by 512,000 +residual, 1.57 GB. 0.4.0 held three of these at once; the new code holds one. ### Measured -Measured with `/usr/bin/time -l` on macOS, on a 6000 by 4000 RGB JPEG (24 million pixels), model Q, -two runs each. "Baseline" loads the models and decodes the image without encoding. +Measured with `/usr/bin/time -l` on macOS (Apple silicon), peak resident memory in MB of 10^6 +bytes, model Q, two runs each, with nothing else running. "Baseline" loads the models and decodes +the image without encoding. + +| Input | Baseline | 0.4.0 `encode` | new `encode` | +| --- | --- | --- | --- | +| 6000 by 4000 RGB JPEG | 221 MB | 1866 to 1875 MB, 0.55 to 0.57 s | 892 to 895 MB, 0.41 s | +| 1 by 2000 RGB PNG | 147 to 150 MB | 4954 to 4957 MB, 3.3 s | 1808 MB, 1.4 s | +| 1 by 4000 RGB PNG | 145 to 146 MB | 9680 MB, 6.4 to 6.5 s | 3389 to 3391 MB, 2.7 s | -| Run | Peak resident memory | Above baseline, per pixel | -| --- | --- | --- | -| Baseline | 223 MB | | -| 0.4.0 `encode` | 1868 MB | 68.5 bytes | -| new `encode` | 589 MB | 15.3 bytes | +Above the baseline, the 6000 by 4000 encode uses 68.5 to 68.9 bytes per pixel in 0.4.0 and 28.0 +to 28.1 bytes per pixel in the new code. -The 15.3 bytes include the 12-byte output and the inference workspace of the encoder model, which -these runs do not separate. Run time for the same image went from 0.57 to 0.48 seconds. The crate's -`cargo bench` (`benches/encode.rs`, a 240 by 240 image) measured 96.7 ms for 0.4.0 and 90.6 ms for -the new code. +The crate's `cargo bench` (`benches/encode.rs`, a 240 by 240 image, dominated by model inference) +measured 92.4 ms (91.8 to 93.0) for 0.4.0 and 93.8 ms (93.0 to 94.8) for the new code, about 1.5% +slower; an earlier run of both gave 94.5 and 95.8 ms. The large images in the table encode faster +with the new code. -Going lower requires an 8-bit result, and `encode` returns 32-bit floating point values. An -additional method that returns `Rgb8` or `Rgba8`, written into the input buffer, would bring the -cost to about 3 bytes per pixel above the decoded image. It is not part of this change. +Two ways to go lower, neither part of this change: + +- Resizing the residual one row at a time avoids the 12-byte full-size residual (measured at 15.4 + bytes per pixel above baseline), at the cost of reproducing the `image` crate's resize. +- An additional method that returns `Rgb8` or `Rgba8`, written into the input buffer, removes the + floating point output as well. + +Memory for very thin images still grows with the ratio of the long side to the short side, about +786 KB per unit, whatever the number of pixels. A service that accepts arbitrary images needs to +limit that ratio itself. Rejecting such images in the crate would change behavior. ## How it was checked -- Unit tests in `image_processing.rs` compare `apply_residual` with the 0.4.0 implementation, kept - in the test module as `apply_residual_reference`, and require identical bytes, color type, - dimensions and color space. They cover ten pixel formats (8-bit, 16-bit and 32-bit, gray and - color, with and without alpha), image sizes larger than, equal to, smaller than and mixed against - the residual, three residual shapes, values outside `[0, 1]`, and residuals outside the range - `encode` produces. -- An end-to-end comparison built 0.4.0 and the new code into one program and ran `encode` with the - real Q and P models on 28 inputs: the sample images, gray, 16-bit and floating point versions of - `ripley.jpg`, and synthetic images from 1x1 to 4000x10. All outputs were identical, as were the - decode results. Two runs of 0.4.0 agreed with each other, so the comparison is not hiding - nondeterminism. -- The same program compared the two versions on NaN, infinite, negative, zero and large strengths, - on images exactly at and one pixel past the 2:1 aspect limit, and on images with Display P3 - primaries and a linear transfer function. All outputs were identical. -- The Python implementation in `python/` decoded all 20 Rust-encoded images (Q and P, from 0.4.0 - and from the new code), and both Rust versions decoded the 6 images encoded by Python. - Unwatermarked images decoded as unwatermarked in both languages. The Python encoder uses a - different algorithm (it removes the per-channel mean of the residual and uses PyTorch's - interpolation), so its output bytes differ from Rust's, as they did with 0.4.0. Image quality is - similar: 41.7 to 44.2 dB PSNR for Rust and 41.8 to 44.6 dB for Python with model Q, 46.8 to - 50.5 dB and 47.0 to 50.8 dB with model P. +Every check was rerun after each code change. + +- During development, unit tests kept the 0.4.0 `apply_residual`, `residual_image`, + `remove_boundary_artifact` and `encode` bodies as reference functions and required identical + values, buffer length, color type, dimensions and color space. They covered: + - `apply_residual` on ten pixel formats, image sizes larger than, equal to, smaller than and + mixed against the residual, three residual shapes, and floating point inputs outside `[0, 1]`; + - `remove_boundary_artifact` and `residual_image` on wide, tall, exactly 2:1, one pixel past + 2:1, square and thin sizes, with NaN, infinite and negative zero residual values, and on + arrays stored in channel order, pixel order, with an offset, with trailing values, with + reversed rows and empty; + - `encode` against the reference `encode` with the real B, C, P and Q models, on seven inputs + and two strengths. + + Those tests failed on changes that only alter rounding: computing `(value + 1) / 2` as + `(value + 1) * 0.3 / 0.6` in `residual_image` made 49,092 values differ, and multiplying the + strength into each side of the subtraction in `encode` made 80,826 values differ. The crate's + tests no longer include the reference code. `residual_image_on_any_layout` checks + `residual_image` against values computed from the array for each of those storage layouts. +- A comparison program built 0.4.0 and the new code into one binary and ran `encode` with the real + Q and P models on 28 inputs (the sample images, gray, 16-bit and floating point versions of + `ripley.jpg`, and synthetic images from 100x60 to 2000x150), and on 112 edge cases: 0x0, 0x10, + 10x0, 1x1, 1x300, 300x1, 1x2000, 2000x1, 4000x10 and 10x4000 images; images exactly at and one + pixel past the 2:1 limit; NaN, infinite, negative, zero, negative zero and large strengths; all + ten pixel formats at three sizes with varying alpha; Display P3 and linear color spaces. Each + version ran each case twice, and both versions were deterministic. All outputs were identical + except the accepted empty-image error, and the program reported a difference when one input bit + was changed. +- The same program ran `encode` on 0x0, 0x10 and 10x0 images with all four variants and with a + valid and an invalid watermark. All outcomes match 0.4.0 except the accepted P and Q error. +- Both versions decoded the 12 images in `images/` and `c2pa/` with each of the B, C, P and Q + models and returned the same result in all 48 cases. +- Five implementations decoded each other's watermarks: Rust 0.4.0, the new Rust code, Python + (`python/`), C++ (`cpp/`, built with OpenCV and ONNX Runtime) and JavaScript (`js/`, unmodified, + in headless Chrome with WebGPU). Rust 0.4.0, the new Rust code, Python and C++ each encoded Q and + P watermarks into five images, 40 files in all. Every Rust, Python and C++ decoder read the + watermark from all 40, and the JavaScript decoder read it from all 40 at full size. The new and + 0.4.0 Rust encodings saved as 8-bit PNG were byte-identical in pairs. +- Thirty sequential encodes of the 9-megapixel `ripley.jpg` held resident memory between 682 and + 684 MB. Eight threads sharing one `Trustmark` ran 48 encodes, each followed by a decode, with no + output differing from the sequential result, no decode failure and no error. +- Independent review passes probed the public interface of 0.4.0 and the new code side by side on + 5,832 further cases (every variant and BCH version, invalid watermark strings, extreme strengths, + all pixel formats at 15 sizes, special floating point pixels, non-sRGB color spaces, encode and + decode), and found no difference other than the accepted one. - `cargo clippy --workspace --all-targets -- -D warnings` and `cargo fmt --check` report nothing, and `cargo test --workspace` passes. ## Behavior to know about -One case changed. `encode` on an image with a width or height of 0 panicked inside `ndarray` in -0.4.0. The new code returns `Error::ImageProcessing(image_processing::Error::Image)` ("invalid image") -before running the model. `decode` on such an image returns `Error::CorruptWatermark`, in 0.4.0 -and in the new code. - -The following cases behave the same in both versions; they are listed because a reader may not -expect them. - -- An image of 100 by 60 pixels encodes, but its watermark does not decode, with either version. -- A residual with an empty dimension makes the internal `apply_residual` return - `Error::InvalidShape`. 0.4.0 resized such a residual to zeros instead. `encode` cannot produce - one: the model returns 256 by 256 values, and the padding only adds to that. +One case changed. With the P and Q variants, `encode` on an image with a width or height of 0 +panicked inside `remove_boundary_artifact` in 0.4.0. The new code returns +`Error::ImageProcessing(image_processing::Error::Image)` ("invalid image") at the same point, +after the watermark is validated and the model has run, so an invalid watermark is still reported +as `Error::Bits`. With the B and C variants such an image returns an empty image, as in 0.4.0. +`decode` on an empty image returns `Error::CorruptWatermark`, in 0.4.0 and in the new code. + +The following cases behave the same in 0.4.0 and in the new code; they are listed because a reader +may not expect them. + +- Thin images encode, but their watermark does not decode. On strips cut from `ripley.jpg` with a + long side of 2000 pixels, Q watermarks decoded from a short side of 96 pixels and P watermarks + from 64. A 2000 by 32 Q strip decoded to a different payload, reported as valid. A 100 by 60 + image also encodes without a decodable watermark. +- Decoders report watermarks in some unwatermarked images. The Rust P decoder reports the same + 75-bit payload for `c2pa/example.jpg` and `images/ufo_240.jpg`, which Python reads as + unwatermarked; the JavaScript decoder reports a payload for both. The Rust and C++ Q decoders + report one for `images/ghost.png` converted to RGB, and the C++ P decoder for `ripley.jpg`. +- Python and JavaScript find no watermark in `c2pa/example_wm.jpg` or + `c2pa/example_wm_signed.jpg`. The Rust B and P decoders report payloads for both, and the Rust Q + decoder reports none. +- The JavaScript example page shrinks images to 500 pixels wide before decoding. After that step it + read a different payload from one of the 40 files, a P watermark encoded by Python into + `ripley.jpg`; at full size it read the right one. diff --git a/rust/src/image_processing.rs b/rust/src/image_processing.rs index 33b51a3..f01bfef 100644 --- a/rust/src/image_processing.rs +++ b/rust/src/image_processing.rs @@ -9,10 +9,10 @@ use std::cmp; use fast_image_resize::{ResizeAlg, ResizeOptions, Resizer}; use image::{ - DynamicImage, GenericImageView as _, GrayAlphaImage, GrayImage, ImageBuffer, Rgb32FImage, - RgbImage, Rgba32FImage, RgbaImage, + imageops::FilterType, DynamicImage, GenericImageView as _, GrayAlphaImage, GrayImage, + ImageBuffer, Pixel, Rgb32FImage, RgbImage, Rgba32FImage, RgbaImage, }; -use ndarray::{s, Array, Array2, Array3, Array4, ArrayD, Axis, ShapeError}; +use ndarray::{s, Array, Array4, ArrayD, Axis, IxDyn, ShapeError}; use ort::value::{TensorValueType, Value}; use crate::Variant; @@ -101,192 +101,77 @@ impl TryFrom> for Array4 { } } -/// Red, green and blue: the channels the residual has and is added to. Alpha gets no residual. -const COLOR_CHANNELS: usize = 3; - -/// Apply `residual` to the `input`. -/// -/// `residual` has shape `[1, 3, height, width]` with values in `[-1, 1]`. It is resized to the -/// size of `input` and added to the color channels; alpha is kept. +/// Converts the residual array, of shape `[1, 3, height, width]`, into an `Rgb32F` image. /// -/// The output equals that of the 0.4.0 implementation (kept as `apply_residual_reference`) bit -/// for bit. The watermark signal is only a few 8-bit steps, so this repeats the arithmetic of -/// `image::imageops::resize` with `FilterType::Triangle` operation for operation, instead of an -/// equivalent formula that rounds differently. Unlike 0.4.0, it makes no full-size copies besides -/// the output: the residual is resized and the input converted one row at a time. -pub(super) fn apply_residual( - input: DynamicImage, - residual: &ArrayD, -) -> Result { - let &[1, COLOR_CHANNELS, residual_height, residual_width] = residual.shape() else { +/// The values are converted in place when the array is stored in pixel order, as +/// `remove_boundary_artifact` stores it, and copied once otherwise. +pub(super) fn residual_image(array: ArrayD) -> Result { + let &[1, 3, height, width] = array.shape() else { return Err(Error::InvalidShape); }; - // The model never produces an empty residual, and an empty one cannot be resized. - if residual_height == 0 || residual_width == 0 { - return Err(Error::InvalidShape); + // `[height, width, 3]`, the pixel order of an image. + let mut pixels = array + .permuted_axes(IxDyn(&[0, 2, 3, 1])) + .remove_axis(Axis(0)); + if !pixels.is_standard_layout() { + pixels = pixels.as_standard_layout().into_owned(); } - // The residual as the 0.4.0 `Rgb32F` image held it: values in `[0, 1]`, indexed - // `[channel, row, column]`. - let residual = Array3::from_shape_fn( - (COLOR_CHANNELS, residual_height, residual_width), - |(channel, row, column)| convert_from_neg1_1_to_0_1!(residual[[0, channel, row, column]]), - ); - - let (width, height) = input.dimensions(); - // `imageops::resize` returns an unchanged copy when the size already matches. - let resize = - ((width as usize, height as usize) != (residual_width, residual_height)).then(|| { - ResizeWeights { - columns: bilinear_resize_weights(residual_width, width), - rows: bilinear_resize_weights(residual_height, height), - } - }); - - // The conversion gives the output the input's color space, as in 0.4.0; the color values - // are overwritten. - Ok(if input.color().has_alpha() { - let mut output = input.to_rgba32f(); - add_residual( - &mut output, - COLOR_CHANNELS + 1, - &input, - &residual, - resize.as_ref(), - ); - output.into() - } else { - let mut output = input.to_rgb32f(); - add_residual( - &mut output, - COLOR_CHANNELS, - &input, - &residual, - resize.as_ref(), - ); - output.into() - }) -} -/// How to resize the residual to the image: one entry per output column and per output row. -struct ResizeWeights { - columns: Vec, - rows: Vec, -} + // Trustmark's residual is in `[-1,1]`. The `image` crate wants `[0,1]`. + pixels.mapv_inplace(|value| convert_from_neg1_1_to_0_1!(value)); -/// How one output position of a resize along one axis is computed: the weighted sum of the -/// input values at `first_input`, `first_input + 1`, and so on, one per weight. -struct WeightedInputs { - first_input: usize, - weights: Vec, -} + // In standard layout the values are contiguous, starting at `offset` in the buffer. + let length = pixels.len(); + let (mut values, offset) = pixels.into_raw_vec_and_offset(); + values.drain(..offset.unwrap_or(0)); + values.truncate(length); -impl WeightedInputs { - /// The weighted sum of the values `input` returns for each position. The additions run in - /// position order, as in `imageops::resize`, since the order changes the rounding. - fn sum(&self, input: impl Fn(usize) -> f32) -> f32 { - let mut sum = 0.0; - for (offset, weight) in self.weights.iter().enumerate() { - sum += input(self.first_input + offset) * weight; - } - sum - } -} + let image = Rgb32FImage::from_vec(width as u32, height as u32, values).ok_or(Error::Image)?; -/// Resizes `input_size` positions to `output_size` positions by blending neighboring values, as -/// `imageops::resize` does with `FilterType::Triangle` (image 0.25, `vertical_sample` and -/// `horizontal_sample`). Computed the same way, so the results round the same way. -fn bilinear_resize_weights(input_size: usize, output_size: u32) -> Vec { - // How many input positions one output position covers. - let scale = input_size as f32 / output_size as f32; - // How far from its center an output position blends inputs: one position when enlarging, - // more when shrinking so that every input counts. - let reach = if scale < 1.0 { 1.0 } else { scale }; - - (0..output_size) - .map(|output_position| { - // Where the center of this output position falls, in input positions. - let center = (output_position as f32 + 0.5) * scale; - let first = ((center - reach).floor() as i64).clamp(0, input_size as i64 - 1); - let end = ((center + reach).ceil() as i64).clamp(first + 1, input_size as i64); - let center = center - 0.5; - - // Nearer inputs count more. Inputs past the edge are left out, and the weights - // rescaled to add up to 1. - let mut weights: Vec = (first..end) - .map(|input_position| triangle((input_position as f32 - center) / reach)) - .collect(); - let total = weights.iter().fold(0.0, |total, weight| total + weight); - for weight in &mut weights { - *weight /= total; - } - WeightedInputs { - first_input: first as usize, - weights, - } - }) - .collect() + Ok(image.into()) } -/// 1 at distance 0, falling in a straight line to 0 at distance 1 and beyond. -fn triangle(distance: f32) -> f32 { - if distance.abs() < 1.0 { - 1.0 - distance.abs() +/// Red, green and blue: the channels the residual has and is added to. Alpha gets no residual. +const COLOR_CHANNELS: usize = 3; + +/// Apply `residual` to the `input`. +/// +/// This function resizes `residual` to the size of `input`, then adds `residual` to the `input`. +/// +/// The residual is freed once resized, and the sum is written into the input converted to +/// `Rgb32F`, or `Rgba32F` when it has alpha, so the resized residual is the only other image held +/// while adding. Converting the input to `Rgb32F` or to `Rgba32F` gives the same color values. +pub(super) fn apply_residual(input: DynamicImage, residual: DynamicImage) -> DynamicImage { + let (w, h) = input.dimensions(); + let resized = residual + .resize_exact(w, h, FilterType::Triangle) + .into_rgb32f(); + drop(residual); + + if input.color().has_alpha() { + let mut output = input.into_rgba32f(); + add_residual(&mut output, &resized); + output.into() } else { - 0.0 + let mut output = input.into_rgb32f(); + add_residual(&mut output, &resized); + output.into() } } -/// Writes `input` plus the `residual` into the color channels of `output`, one row at a time. -/// `output` holds `channels` values per pixel. The residual is resized with `resize`, or used -/// as is when that is `None`. -fn add_residual( - output: &mut [f32], - channels: usize, - input: &DynamicImage, - residual: &Array3, - resize: Option<&ResizeWeights>, +/// Adds `residual` to the color channels of `image`. +fn add_residual>( + image: &mut ImageBuffer>, + residual: &Rgb32FImage, ) { - let width = input.width(); - let residual_width = residual.dim().2; - // One row of the residual, resized to the output height but not yet to the output width. - let mut residual_row = Array2::zeros((COLOR_CHANNELS, residual_width)); - - for (row, output_row) in output - .chunks_exact_mut(width as usize * channels) - .enumerate() - { - // Resize vertically first, as `imageops::resize` does (`vertical_sample`). - if let Some(resize) = resize { - for channel in 0..COLOR_CHANNELS { - for column in 0..residual_width { - residual_row[[channel, column]] = - resize.rows[row].sum(|source_row| residual[[channel, source_row, column]]); - } - } - } - - // 0.4.0 read the input converted to `Rgba32F`, whatever the output type. - let input_row = input.crop_imm(0, row as u32, width, 1).into_rgba32f(); - - for (column, output_pixel) in output_row.chunks_exact_mut(channels).enumerate() { - let input_pixel = input_row.get_pixel(column as u32, 0); - for channel in 0..COLOR_CHANNELS { - let residual_value = match resize { - // Then horizontally (`horizontal_sample`), which limits the result to - // `[0, 1]`. - Some(resize) => resize.columns[column] - .sum(|source_column| residual_row[[channel, source_column]]) - .clamp(0.0, 1.0), - None => residual[[channel, row, column]], - }; - - // The 0.4.0 `apply_residual` closure: both values to `[-1, 1]`, added, capped - // at 1, and back to `[0, 1]`. - let pixel_value = convert_from_0_1_to_neg1_1!(input_pixel[channel]); - let residual_value = convert_from_0_1_to_neg1_1!(residual_value); - output_pixel[channel] = - convert_from_neg1_1_to_0_1!(f32::min(pixel_value + residual_value, 1.0)); - } + for (pixel, residual_pixel) in image.pixels_mut().zip(residual.pixels()) { + let values = pixel.channels_mut(); + for channel in 0..COLOR_CHANNELS { + // Both values to `[-1, 1]`, added, capped at 1, and back to `[0, 1]`. + let pixel_value = convert_from_0_1_to_neg1_1!(values[channel]); + let residual_value = convert_from_0_1_to_neg1_1!(residual_pixel[channel]); + values[channel] = + convert_from_neg1_1_to_0_1!(f32::min(pixel_value + residual_value, 1.0)); } } } @@ -367,15 +252,16 @@ pub(super) fn remove_boundary_artifact( .map(|i| residual.slice(s![.., i, .., ..]).mean().unwrap()) .collect(); - // We want one dimension of the output to be 256 and we we want the aspect ratio of the output - // to match the input image. + // We want one dimension of the output to be 256 and we want the aspect ratio of the output + // to match the input image. The array has shape `[1, 3, height, width]` but is stored in pixel + // order, `[1, height, width, 3]`, so `residual_image` needs no copy of it. let other_dim; let mut mean_padded: ndarray::Array4 = if width > height { other_dim = ((width as f32 / height as f32) * 256.0) as usize; - ndarray::Array4::zeros([1, 3, 256_usize, other_dim]) + ndarray::Array4::zeros([1, 256_usize, other_dim, 3]).permuted_axes([0, 3, 1, 2]) } else { other_dim = ((height as f32 / width as f32) * 256.0) as usize; - ndarray::Array4::zeros([1, 3, other_dim, 256]) + ndarray::Array4::zeros([1, other_dim, 256, 3]).permuted_axes([0, 3, 1, 2]) }; // This softens the transition between the residual area and the rest of the image. @@ -405,73 +291,8 @@ pub(super) fn remove_boundary_artifact( #[cfg(test)] mod tests { - use image::{ - imageops::{self, FilterType}, - Pixel as _, - }; - use super::*; - /// The 0.4.0 conversion of the residual array into an `Rgb32F` image, kept as the reference - /// for `apply_residual`. - fn residual_image_reference(mut array: ArrayD) -> Result { - let &[1, 3, height, width] = &array.shape().to_owned()[..] else { - return Err(Error::InvalidShape); - }; - array.swap_axes(2, 3); - let array = array.reversed_axes().remove_axis(Axis(3)); - let array = array.to_shape([width * height * 3])?; - - // The `image` crate normalizes to `[0,1]`. Trustmark wants images normalized to `[-1,1]`. - let array = convert_from_neg1_1_to_0_1!(array); - - let image = Rgb32FImage::from_vec(width as u32, height as u32, array.to_vec()) - .ok_or(Error::Image)?; - - Ok(image.into()) - } - - /// The 0.4.0 `apply_residual`, kept as the reference for `apply_residual`. - fn apply_residual_reference(input: DynamicImage, residual: DynamicImage) -> DynamicImage { - let has_alpha = input.color().has_alpha(); - let (w, h) = input.dimensions(); - - let applied = { - let input = input.clone().into_rgba32f(); - let mut target = input.clone(); - - let residual = residual.resize_exact(w, h, FilterType::Triangle); - let residual = residual.into_rgba32f(); - - for ((target, residual), original) in target - .pixels_mut() - .zip(residual.pixels()) - .zip(input.pixels()) - { - target.apply2(residual, |x, y| { - let x = convert_from_0_1_to_neg1_1!(x); - let y = convert_from_0_1_to_neg1_1!(y); - - convert_from_neg1_1_to_0_1!(f32::min(x + y, 1.0)) - }); - target[3] = original[3]; - } - - target - }; - - if has_alpha { - let mut input = input.into_rgba32f(); - imageops::replace(&mut input, &applied, 0, 0); - input.into() - } else { - let mut input = input.into_rgb32f(); - let applied = DynamicImage::ImageRgba32F(applied).into_rgb32f(); - imageops::replace(&mut input, &applied, 0, 0); - input.into() - } - } - #[test] fn renormalize_from_0_1() { assert_eq!(convert_from_0_1_to_neg1_1!(0.), -1.); @@ -533,127 +354,43 @@ mod tests { assert_eq!(output.shape(), &[1, 3, 298, 256]); } - /// Deterministic values in `[0, 1)` (xorshift), so the tests need no random number crate. - fn noise(mut state: u32) -> impl FnMut() -> f32 { - move || { - state ^= state << 13; - state ^= state >> 17; - state ^= state << 5; - (state >> 8) as f32 / (1 << 24) as f32 - } - } - - fn residual(height: usize, width: usize, seed: u32) -> ArrayD { - let mut noise = noise(seed); - Array4::from_shape_simple_fn([1, 3, height, width], || noise() * 0.4 - 0.2).into_dyn() - } - - fn noise_image(width: u32, height: u32, seed: u32) -> Rgba32FImage { - let mut noise = noise(seed); - Rgba32FImage::from_fn(width, height, |_, _| { - image::Rgba([noise(), noise(), noise(), noise()]) - }) - } - - /// `apply_residual` output against the 0.4.0 implementation, bit for bit. - fn assert_matches_reference(input: &DynamicImage, residual: &ArrayD) { - let actual = apply_residual(input.clone(), residual).unwrap(); - let expected = apply_residual_reference( - input.clone(), - residual_image_reference(residual.clone()).unwrap(), - ); - assert_eq!(actual.color(), expected.color()); - assert_eq!(actual.dimensions(), expected.dimensions()); - assert_eq!(actual.color_space(), expected.color_space()); - // Both are `f32` images; compare each value's bytes. - let (actual, expected) = ( - actual.as_bytes().as_chunks::<4>().0, - expected.as_bytes().as_chunks::<4>().0, - ); - let mismatches = actual.iter().zip(expected).filter(|(a, e)| a != e).count(); - assert_eq!( - mismatches, - 0, - "{:?} {}x{}, residual {:?}: {mismatches} of {} values differ", - input.color(), - input.width(), - input.height(), - residual.shape(), - expected.len() - ); - } - - const RESIDUAL_SHAPES: [(usize, usize); 3] = [(256, 256), (256, 384), (384, 256)]; - - #[test] - fn apply_residual_matches_reference_on_sample_images() { - for path in ["../images/ghost.png", "../images/ufo_240.jpg"] { - let img = image::open(path).unwrap(); - for (seed, (h, w)) in RESIDUAL_SHAPES.into_iter().enumerate() { - assert_matches_reference(&img, &residual(h, w, seed as u32 + 1)); - } - } - } - - #[test] - fn apply_residual_matches_reference_on_every_color_type() { - // Larger than, equal to, smaller than and mixed against the residual sizes. - let sizes = [(600, 450), (256, 256), (256, 300), (100, 60), (200, 500)]; - for (i, (w, h)) in sizes.into_iter().enumerate() { - let float = DynamicImage::ImageRgba32F(noise_image(w, h, 100 + i as u32)); - let inputs = [ - DynamicImage::ImageLuma8(float.to_luma8()), - DynamicImage::ImageLumaA8(float.to_luma_alpha8()), - DynamicImage::ImageRgb8(float.to_rgb8()), - DynamicImage::ImageRgba8(float.to_rgba8()), - DynamicImage::ImageLuma16(float.to_luma16()), - DynamicImage::ImageLumaA16(float.to_luma_alpha16()), - DynamicImage::ImageRgb16(float.to_rgb16()), - DynamicImage::ImageRgba16(float.to_rgba16()), - DynamicImage::ImageRgb32F(float.to_rgb32f()), - float, - ]; - for input in &inputs { - for (seed, (rh, rw)) in RESIDUAL_SHAPES.into_iter().enumerate() { - assert_matches_reference(input, &residual(rh, rw, seed as u32 + 10)); - } - } - } - } - #[test] - fn apply_residual_matches_reference_on_out_of_range_floats() { - let mut noise = noise(7); - let img = Rgb32FImage::from_fn(300, 280, |_, _| { - image::Rgb([ - noise() * 3.0 - 1.0, - noise() * 3.0 - 1.0, - noise() * 3.0 - 1.0, - ]) - }); - assert_matches_reference(&DynamicImage::ImageRgb32F(img), &residual(256, 256, 3)); - } - - #[test] - fn apply_residual_matches_reference_on_out_of_range_residual() { - // `encode` limits the residual to `[-0.2, 0.2]`; `apply_residual` must still match 0.4.0 - // beyond that, with and without resizing. - let residual = residual(256, 256, 4).mapv(|value| value * 8.0); - for (w, h) in [(256, 256), (300, 280)] { - let img = DynamicImage::ImageRgba32F(noise_image(w, h, 9)); - assert_matches_reference(&img, &residual); - } - } - - #[test] - fn apply_residual_rejects_bad_shape() { - for shape in [[1, 4, 256, 256], [1, 3, 0, 256], [1, 3, 256, 0]] { - let img = DynamicImage::ImageRgb8(RgbImage::new(10, 10)); - let residual = Array4::::zeros(shape).into_dyn(); - assert!(matches!( - apply_residual(img, &residual), - Err(Error::InvalidShape) - )); + fn residual_image_on_any_layout() { + // Every value differs, so values read from the wrong place change the image. + let value = |n: usize, c: usize, y: usize, x: usize| { + ((n * 3 + c) * 600 + y * 30 + x) as f32 / 4000.0 - 0.5 + }; + // Channel order, as the model returns it, is copied once. Pixel order, as + // `remove_boundary_artifact` stores it, is converted in place, also when the values start + // at an offset in the buffer or are followed by other values. + let channel_order = Array4::from_shape_fn([1, 3, 20, 30], |(n, c, y, x)| value(n, c, y, x)); + let mut reversed_rows = channel_order.clone(); + reversed_rows.invert_axis(Axis(2)); + let pixel_order = Array4::from_shape_fn([2, 20, 30, 3], |(n, y, x, c)| value(n, c, y, x)); + let mut at_offset = pixel_order.clone(); + at_offset.slice_collapse(s![1.., .., .., ..]); + let mut with_trailing_values = pixel_order; + with_trailing_values.slice_collapse(s![..1, .., .., ..]); + let arrays = [ + channel_order, + reversed_rows, + at_offset.permuted_axes([0, 3, 1, 2]), + with_trailing_values.permuted_axes([0, 3, 1, 2]), + Array4::zeros([1, 3, 0, 30]), + ]; + for array in arrays { + let (height, width) = (array.shape()[2], array.shape()[3]); + let expected = Rgb32FImage::from_fn(width as u32, height as u32, |x, y| { + image::Rgb(std::array::from_fn(|c| { + convert_from_neg1_1_to_0_1!(array[[0, c, y as usize, x as usize]]) + })) + }); + let actual = residual_image(array.into_dyn()).unwrap(); + assert_eq!(actual, DynamicImage::ImageRgb32F(expected)); } + assert!(matches!( + residual_image(Array4::zeros([1, 4, 2, 2]).into_dyn()), + Err(Error::InvalidShape) + )); } } diff --git a/rust/src/lib.rs b/rust/src/lib.rs index 63dde74..a666948 100644 --- a/rust/src/lib.rs +++ b/rust/src/lib.rs @@ -120,7 +120,7 @@ impl Trustmark { /// which will be watermarked. `strength` is a number between 0 and 1 indicating how strong the /// resulting watermark should be. 0.95 is a normal strength. /// - /// An image with a width or height of 0 is an error. + /// With the P and Q variants, an image with a width or height of 0 is an error. pub fn encode( &self, watermark: String, @@ -128,9 +128,6 @@ impl Trustmark { strength: f32, ) -> Result { let (original_width, original_height) = img.dimensions(); - if original_width == 0 || original_height == 0 { - return Err(image_processing::Error::Image.into()); - } let aspect_ratio = original_width as f32 / original_height as f32; // the image is always encoded with size 256x256 @@ -140,7 +137,8 @@ impl Trustmark { // crop and resize. let model_input: ndarray::Array4 = ModelImage(encode_size, self.variant, &img).try_into()?; - // Same error variant as when the `ModelImage` conversion created the tensor. + // A failure is reported as an image processing error, as `TryFrom for Value` + // reports it in `decode`. let input_img: Value> = Value::from_array(model_input.clone()).map_err(image_processing::Error::from)?; let bits: Value> = @@ -166,6 +164,10 @@ impl Trustmark { if (self.variant == Variant::Q && !(0.5..=2.0).contains(&aspect_ratio)) || self.variant == Variant::P { + // `remove_boundary_artifact` cannot pad to the proportions of an empty image. + if original_width == 0 || original_height == 0 { + return Err(image_processing::Error::Image.into()); + } residual = image_processing::remove_boundary_artifact( residual, (original_width as usize, original_height as usize), @@ -173,7 +175,9 @@ impl Trustmark { ); } - Ok(image_processing::apply_residual(img, &residual)?) + let residual = image_processing::residual_image(residual)?; + + Ok(image_processing::apply_residual(img, residual)) } /// Decode a watermark from an image. @@ -222,16 +226,23 @@ mod tests { } #[test] - fn encode_rejects_empty_image() { - for variant in [Variant::Q, Variant::P] { + fn encode_on_empty_image() { + for variant in [Variant::B, Variant::C, Variant::P, Variant::Q] { let tm = Trustmark::new("./models", variant, Version::Bch5).unwrap(); for (width, height) in [(0, 0), (0, 10), (10, 0)] { let input = DynamicImage::new_rgb8(width, height); - let result = tm.encode("0010101".to_owned(), input, 0.95); - assert!(matches!( - result, - Err(Error::ImageProcessing(image_processing::Error::Image)) - )); + let result = tm.encode("0010101".to_owned(), input.clone(), 0.95); + if matches!(variant, Variant::P | Variant::Q) { + assert!(matches!( + result, + Err(Error::ImageProcessing(image_processing::Error::Image)) + )); + } else { + assert_eq!(result.unwrap().dimensions(), (width, height)); + } + // An invalid watermark is reported before the image size. + let result = tm.encode("not bits".to_owned(), input, 0.95); + assert!(matches!(result, Err(Error::Bits(_)))); } } } From 2d6a4de4e49ab3cf894f8f3c0b28558414c52884 Mon Sep 17 00:00:00 2001 From: Tania Mathern Date: Thu, 5 Nov 2026 14:21:09 -0800 Subject: [PATCH 3/4] fix: WIP 2 --- rust/src/image_processing.rs | 192 +++++++++++++++++++++++++++++++++++ rust/src/lib.rs | 5 +- 2 files changed, 195 insertions(+), 2 deletions(-) diff --git a/rust/src/image_processing.rs b/rust/src/image_processing.rs index f01bfef..d4f97aa 100644 --- a/rust/src/image_processing.rs +++ b/rust/src/image_processing.rs @@ -291,8 +291,200 @@ pub(super) fn remove_boundary_artifact( #[cfg(test)] mod tests { + use image::{ + imageops, + metadata::{CicpColorPrimaries, CicpTransferCharacteristics}, + }; + use super::*; + fn residual_image_main_reference(mut array: ArrayD) -> DynamicImage { + let &[1, 3, height, width] = array.shape() else { + panic!("invalid reference residual shape"); + }; + array.swap_axes(2, 3); + let array = array.reversed_axes().remove_axis(Axis(3)); + let array = array.to_shape([width * height * 3]).unwrap(); + let array = (array + 1.0_f32) / 2.0_f32; + Rgb32FImage::from_vec(width as u32, height as u32, array.to_vec()) + .unwrap() + .into() + } + + fn apply_residual_main_reference(input: DynamicImage, residual: DynamicImage) -> DynamicImage { + let has_alpha = input.color().has_alpha(); + let (width, height) = input.dimensions(); + let applied = { + let input = input.clone().into_rgba32f(); + let mut target = input.clone(); + let residual = residual + .resize_exact(width, height, FilterType::Triangle) + .into_rgba32f(); + for ((target, residual), original) in target + .pixels_mut() + .zip(residual.pixels()) + .zip(input.pixels()) + { + target.apply2(residual, |pixel, residual| { + let pixel = pixel * 2.0_f32 - 1.0_f32; + let residual = residual * 2.0_f32 - 1.0_f32; + (f32::min(pixel + residual, 1.0_f32) + 1.0_f32) / 2.0_f32 + }); + target[3] = original[3]; + } + target + }; + if has_alpha { + let mut output = input.into_rgba32f(); + imageops::replace(&mut output, &applied, 0, 0); + output.into() + } else { + let mut output = input.into_rgb32f(); + let applied = DynamicImage::ImageRgba32F(applied).into_rgb32f(); + imageops::replace(&mut output, &applied, 0, 0); + output.into() + } + } + + fn assert_composition_matches_main(input: &DynamicImage, residual: &ArrayD) { + let expected = apply_residual_main_reference( + input.clone(), + residual_image_main_reference(residual.clone()), + ); + let actual = apply_residual(input.clone(), residual_image(residual.clone()).unwrap()); + assert_eq!(actual.dimensions(), input.dimensions()); + assert_eq!(actual.color(), expected.color()); + assert_eq!(actual.color_space(), input.color_space()); + assert_eq!(actual.color_space(), expected.color_space()); + assert_eq!(actual.as_bytes().len(), expected.as_bytes().len()); + let actual_values = actual.as_bytes().as_chunks::<4>().0; + let expected_values = expected.as_bytes().as_chunks::<4>().0; + for (index, (actual, expected)) in actual_values.iter().zip(expected_values).enumerate() { + assert_eq!( + actual, + expected, + "float {index}, input {:?} {:?}, residual {:?}", + input.color(), + input.dimensions(), + residual.shape() + ); + } + if input.color().has_alpha() { + let original = input.to_rgba32f(); + for (actual, original) in actual.as_rgba32f().unwrap().pixels().zip(original.pixels()) { + assert_eq!(actual[3].to_bits(), original[3].to_bits()); + } + } + } + + fn composition_residual(height: usize, width: usize) -> ArrayD { + Array4::from_shape_fn([1, 3, height, width], |(_, channel, row, column)| { + ((channel * 1027 + row * 131 + column * 97) % 401) as f32 / 1000.0 - 0.2 + }) + .into_dyn() + } + + #[test] + fn apply_residual_matches_main_on_every_color_type() { + for (width, height) in [(48, 35), (32, 32), (32, 40), (16, 10), (20, 50)] { + let float = + DynamicImage::ImageRgba32F(Rgba32FImage::from_fn(width, height, |column, row| { + image::Rgba([ + ((column * 17 + row * 13) % 256) as f32 / 255.0, + ((column * 7 + row * 19) % 256) as f32 / 255.0, + ((column * 23 + row * 3) % 256) as f32 / 255.0, + ((column * 11 + row * 29) % 256) as f32 / 255.0, + ]) + })); + let inputs = [ + DynamicImage::ImageLuma8(float.to_luma8()), + DynamicImage::ImageLumaA8(float.to_luma_alpha8()), + DynamicImage::ImageRgb8(float.to_rgb8()), + DynamicImage::ImageRgba8(float.to_rgba8()), + DynamicImage::ImageLuma16(float.to_luma16()), + DynamicImage::ImageLumaA16(float.to_luma_alpha16()), + DynamicImage::ImageRgb16(float.to_rgb16()), + DynamicImage::ImageRgba16(float.to_rgba16()), + DynamicImage::ImageRgb32F(float.to_rgb32f()), + float, + ]; + for input in inputs { + let default = input.color_space(); + let color_spaces = [ + default, + image::metadata::Cicp { + primaries: CicpColorPrimaries::SmpteRp432, + ..default + }, + image::metadata::Cicp { + transfer: CicpTransferCharacteristics::Linear, + ..default + }, + ]; + for color_space in color_spaces { + let mut input = input.clone(); + input.set_color_space(color_space).unwrap(); + for (height, width) in [(32, 32), (32, 48), (48, 32)] { + assert_composition_matches_main( + &input, + &composition_residual(height, width), + ); + } + } + } + } + } + + #[test] + fn apply_residual_matches_main_on_sample_images() { + for path in ["../images/ghost.png", "../images/ufo_240.jpg"] { + let input = image::open(path).unwrap(); + for (height, width) in [(256, 256), (256, 384), (384, 256)] { + assert_composition_matches_main(&input, &composition_residual(height, width)); + } + } + } + + #[test] + fn apply_residual_matches_main_on_special_floats() { + let values = [ + -1.0, + -0.0, + 0.0, + 0.5, + 1.0, + 2.0, + f32::MIN_POSITIVE, + f32::from_bits(1), + f32::MIN, + f32::MAX, + f32::INFINITY, + f32::NEG_INFINITY, + f32::NAN, + f32::from_bits(0xffc01234), + ]; + for (width, height) in [(32, 32), (48, 35), (16, 10)] { + let float = + DynamicImage::ImageRgba32F(Rgba32FImage::from_fn(width, height, |column, row| { + image::Rgba(std::array::from_fn(|channel| { + values[(column as usize + row as usize * 7 + channel * 3) % values.len()] + })) + })); + for input in [DynamicImage::ImageRgb32F(float.to_rgb32f()), float] { + for (height, width) in [(32, 32), (32, 48), (48, 32)] { + let residual = Array4::from_shape_fn( + [1, 3, height, width], + |(_, channel, row, column)| { + values[(row * 3 + column + channel * 5) % values.len()] + }, + ) + .into_dyn(); + assert_composition_matches_main(&input, &residual); + } + } + } + } + #[test] fn renormalize_from_0_1() { assert_eq!(convert_from_0_1_to_neg1_1!(0.), -1.); diff --git a/rust/src/lib.rs b/rust/src/lib.rs index a666948..b319ee9 100644 --- a/rust/src/lib.rs +++ b/rust/src/lib.rs @@ -185,15 +185,16 @@ impl Trustmark { // P variant has a smaller decode size let decode_size = if self.variant == Variant::P { 224 } else { 256 }; - let img: Value> = + let input_img: Value> = ModelImage(decode_size, self.variant, &img).try_into()?; + drop(img); let watermark = { let mut decoder = self .decoder .lock() .map_err(|_| Error::SessionLockPoisoned)?; let outputs = decoder.run(ort::inputs![ - "image" => img, + "image" => input_img, ])?; outputs["output"].try_extract_array::()?.to_owned() }; From a65b19968d21937c1bf1e0364fcfae8b5db00791 Mon Sep 17 00:00:00 2001 From: Tania Mathern Date: Thu, 5 Nov 2026 15:29:02 -0800 Subject: [PATCH 4/4] fix: A clean up --- impl-notes.md | 250 -------------------------------------------------- 1 file changed, 250 deletions(-) delete mode 100644 impl-notes.md diff --git a/impl-notes.md b/impl-notes.md deleted file mode 100644 index 9f1b3b0..0000000 --- a/impl-notes.md +++ /dev/null @@ -1,250 +0,0 @@ -# Lower peak memory in the Rust `Trustmark::encode` - -This note explains a change to the Rust crate in `rust/`: what was changed, the arithmetic the code -performs, and where the memory saving comes from. It is written for engineers who know Rust but -not image processing. - -## What changed and what did not - -Two files changed: `rust/src/image_processing.rs` and `rust/src/lib.rs`. - -The public interface of the crate is unchanged. `Trustmark::new`, `Trustmark::encode` and -`Trustmark::decode` keep their signatures, and the error enums keep their variants. `encode` -still returns an `Rgb32F` image, or `Rgba32F` when the input has an alpha channel, with the same -color space metadata as before. Every value in the returned image is bit-for-bit identical to what -trustmark 0.4.0 returns for the same input, strength and model. - -The changes are internal: - -- `ModelImage` borrows the input image instead of owning a copy, which removes the two - `img.clone()` calls in `encode`. -- `encode` crops and resizes the input to the 256x256 model input once and reuses that array for - the residual; 0.4.0 did it twice for the same result. -- `remove_boundary_artifact` stores the padded residual in pixel order, and `residual_image` - converts it into an image in place. 0.4.0 held three copies of the padded residual at once. -- `apply_residual` writes the result into the input converted to `Rgb32F` (or `Rgba32F` with - alpha). 0.4.0 made two `Rgba32F` copies of the input, converted the resized residual to - `Rgba32F`, and copied the result into a third buffer at the end. The residual is still resized - by the same `image` call as in 0.4.0. -- With the P and Q variants, `encode` returns an error for an image with a width or height of 0, - where 0.4.0 panicked. - -## How a watermark is added - -### Pixel values and the residual - -An image is a grid of pixels, and each pixel has a red, a green and a blue value (and sometimes an -alpha value for transparency, which the watermark never touches). The code works with these values -as numbers between 0 and 1: 0 is no light, 1 is full intensity. - -The watermarking model only accepts images of 256 by 256 pixels. `encode` shrinks the input to that -size, runs the model, and subtracts the model's input from its output. The difference is the -residual: a 256 by 256 grid of small corrections, one per channel, which encode the watermark. The -code multiplies it by the requested strength and limits each correction to plus or minus 0.2. - -This is Algorithm 1 of the TrustMark paper (arXiv 2311.18297): compute the residual at the model's -size, enlarge it to the image's size, add it to the original image. - -### Padding the residual - -For the P variant, and for Q images more than twice as wide as tall (or the reverse), the model -only saw a square cut from the middle of the image. `remove_boundary_artifact` then builds a -larger residual with the proportions of the whole image: the 256 by 256 residual sits in the -middle, and the rest is filled with the residual's average value for each channel. For a 1000 by -250 image this padded residual is 1024 by 256. - -The padded residual is an array with the shape `[1, 3, height, width]`: all red values first, then -all green, then all blue. An image buffer stores the three values of each pixel next to each -other. 0.4.0 allocated the array in the first order and copied it twice while converting it into -an image. The array now keeps the same shape and the same values, but its memory is laid out in -pixel order, so `residual_image` converts each value from `[-1, 1]` to `[0, 1]` in place and hands -the buffer to the image without copying it. Arrays in the other order, such as the unpadded 256 by -256 residual, are copied once. - -### Enlarging the residual - -The residual has to be stretched to the size of the real image, for example from 256 to 6000 -columns. Each column of the large image falls somewhere between two columns of the small residual, -and its value is a blend of those two, weighted by distance. A position 30% of the way from column -4 to column 5 gets 70% of column 4's value and 30% of column 5's. Positions past the first or last -column take the edge value. Rows are handled the same way. This is bilinear interpolation, which -`image::imageops::resize` performs with `FilterType::Triangle`. Both 0.4.0 and the new code call it -through `DynamicImage::resize_exact`. - -### Adding it to the image - -For each pixel and each color channel, 0.4.0 computed, in 32-bit floating point: - -```text -x = 2 * pixel - 1 pixel value moved from [0, 1] to [-1, 1] -r = 2 * residual - 1 residual value moved from [0, 1] to [-1, 1] -out = (min(x + r, 1) + 1) / 2 -``` - -The new code runs the same three lines, in the same order, on each pixel of the converted input, -and overwrites the color values in place. Alpha values are not written. 0.4.0 also left alpha -unchanged: it computed the formula on all four channels and then copied the original alpha back. - -### Why the output is bit-for-bit identical - -The watermark changes a typical pixel by only a few 8-bit steps; the figures in the paper show -residuals scaled up 20 and 100 times to make them visible. Floating point results depend on the -order of operations, and after conversion to 8 bits a rounding difference can move a pixel by one -step, a noticeable part of that signal. So the change keeps every floating point operation of -0.4.0: - -- The padded residual holds the same values; only the order of the values in memory changed. -- The conversion of the residual to `[0, 1]` is the same `(value + 1) / 2` on each value. A probe - ran all 4,294,967,296 possible 32-bit floating point values, NaN and infinities included, - through the 0.4.0 conversion and the new one, in both memory orders, and found no differing bit. -- The residual is resized by the same `image` call. -- The input is converted by the `image` crate. With alpha, this is the same `Rgba32F` conversion - 0.4.0 used. Without alpha, the new code converts to `Rgb32F`, where 0.4.0 read from an - `Rgba32F` copy. The two conversions give the same color values: a probe compared them bit for - bit for every 8-bit and 16-bit value in all ten pixel formats, and for 4,006 floating point - values including NaN and infinities, in sRGB, Display P3 and linear color spaces, and found no - difference. -- The sum is the 0.4.0 formula. - -An earlier version of this change also replaced `imageops::resize` with code that produced the -resized residual one row at a time. It saved about 13 more bytes per pixel and was bit-identical, -but it reproduced the internals of the `image` crate. It was dropped in favor of calling the crate. - -## Memory - -### Where 0.4.0 spent it - -For an input of N pixels, 0.4.0 allocated, among others: - -- two copies of the input image in `encode`, used only to shrink them to 256 by 256; -- three copies of the padded residual while converting it into an image; -- in `apply_residual`, two 32-bit RGBA copies of the input (16 bytes per pixel each); -- the residual enlarged to full size as 32-bit RGB (12 bytes per pixel), then converted to RGBA - (16 bytes per pixel); -- at the end, a 32-bit RGB copy of the input and a 32-bit RGB copy of the result. - -Several of these are alive at the same moment, on top of the decoded input. - -### Where the new code spends it - -- The input image, as the caller decoded it, consumed by `encode` (3 bytes per pixel for an - ordinary RGB JPEG). It is freed once converted. -- One copy of the padded residual, freed once resized. -- The residual enlarged to full size as 32-bit RGB (12 bytes per pixel). -- The returned image (12 bytes per pixel for RGB, 16 with alpha), a size set by the public return - type. -- The intermediate buffer of the resize, 16 bytes per value for the residual's width times the - image's height. - -The padded residual has 256 by (256 times the long side divided by the short side) values per -channel, about 786 KB for each unit of that ratio. For most photographs it is a few megabytes. -For very thin images it is larger than the image itself: a 1 by 2000 image gets a 256 by 512,000 -residual, 1.57 GB. 0.4.0 held three of these at once; the new code holds one. - -### Measured - -Measured with `/usr/bin/time -l` on macOS (Apple silicon), peak resident memory in MB of 10^6 -bytes, model Q, two runs each, with nothing else running. "Baseline" loads the models and decodes -the image without encoding. - -| Input | Baseline | 0.4.0 `encode` | new `encode` | -| --- | --- | --- | --- | -| 6000 by 4000 RGB JPEG | 221 MB | 1866 to 1875 MB, 0.55 to 0.57 s | 892 to 895 MB, 0.41 s | -| 1 by 2000 RGB PNG | 147 to 150 MB | 4954 to 4957 MB, 3.3 s | 1808 MB, 1.4 s | -| 1 by 4000 RGB PNG | 145 to 146 MB | 9680 MB, 6.4 to 6.5 s | 3389 to 3391 MB, 2.7 s | - -Above the baseline, the 6000 by 4000 encode uses 68.5 to 68.9 bytes per pixel in 0.4.0 and 28.0 -to 28.1 bytes per pixel in the new code. - -The crate's `cargo bench` (`benches/encode.rs`, a 240 by 240 image, dominated by model inference) -measured 92.4 ms (91.8 to 93.0) for 0.4.0 and 93.8 ms (93.0 to 94.8) for the new code, about 1.5% -slower; an earlier run of both gave 94.5 and 95.8 ms. The large images in the table encode faster -with the new code. - -Two ways to go lower, neither part of this change: - -- Resizing the residual one row at a time avoids the 12-byte full-size residual (measured at 15.4 - bytes per pixel above baseline), at the cost of reproducing the `image` crate's resize. -- An additional method that returns `Rgb8` or `Rgba8`, written into the input buffer, removes the - floating point output as well. - -Memory for very thin images still grows with the ratio of the long side to the short side, about -786 KB per unit, whatever the number of pixels. A service that accepts arbitrary images needs to -limit that ratio itself. Rejecting such images in the crate would change behavior. - -## How it was checked - -Every check was rerun after each code change. - -- During development, unit tests kept the 0.4.0 `apply_residual`, `residual_image`, - `remove_boundary_artifact` and `encode` bodies as reference functions and required identical - values, buffer length, color type, dimensions and color space. They covered: - - `apply_residual` on ten pixel formats, image sizes larger than, equal to, smaller than and - mixed against the residual, three residual shapes, and floating point inputs outside `[0, 1]`; - - `remove_boundary_artifact` and `residual_image` on wide, tall, exactly 2:1, one pixel past - 2:1, square and thin sizes, with NaN, infinite and negative zero residual values, and on - arrays stored in channel order, pixel order, with an offset, with trailing values, with - reversed rows and empty; - - `encode` against the reference `encode` with the real B, C, P and Q models, on seven inputs - and two strengths. - - Those tests failed on changes that only alter rounding: computing `(value + 1) / 2` as - `(value + 1) * 0.3 / 0.6` in `residual_image` made 49,092 values differ, and multiplying the - strength into each side of the subtraction in `encode` made 80,826 values differ. The crate's - tests no longer include the reference code. `residual_image_on_any_layout` checks - `residual_image` against values computed from the array for each of those storage layouts. -- A comparison program built 0.4.0 and the new code into one binary and ran `encode` with the real - Q and P models on 28 inputs (the sample images, gray, 16-bit and floating point versions of - `ripley.jpg`, and synthetic images from 100x60 to 2000x150), and on 112 edge cases: 0x0, 0x10, - 10x0, 1x1, 1x300, 300x1, 1x2000, 2000x1, 4000x10 and 10x4000 images; images exactly at and one - pixel past the 2:1 limit; NaN, infinite, negative, zero, negative zero and large strengths; all - ten pixel formats at three sizes with varying alpha; Display P3 and linear color spaces. Each - version ran each case twice, and both versions were deterministic. All outputs were identical - except the accepted empty-image error, and the program reported a difference when one input bit - was changed. -- The same program ran `encode` on 0x0, 0x10 and 10x0 images with all four variants and with a - valid and an invalid watermark. All outcomes match 0.4.0 except the accepted P and Q error. -- Both versions decoded the 12 images in `images/` and `c2pa/` with each of the B, C, P and Q - models and returned the same result in all 48 cases. -- Five implementations decoded each other's watermarks: Rust 0.4.0, the new Rust code, Python - (`python/`), C++ (`cpp/`, built with OpenCV and ONNX Runtime) and JavaScript (`js/`, unmodified, - in headless Chrome with WebGPU). Rust 0.4.0, the new Rust code, Python and C++ each encoded Q and - P watermarks into five images, 40 files in all. Every Rust, Python and C++ decoder read the - watermark from all 40, and the JavaScript decoder read it from all 40 at full size. The new and - 0.4.0 Rust encodings saved as 8-bit PNG were byte-identical in pairs. -- Thirty sequential encodes of the 9-megapixel `ripley.jpg` held resident memory between 682 and - 684 MB. Eight threads sharing one `Trustmark` ran 48 encodes, each followed by a decode, with no - output differing from the sequential result, no decode failure and no error. -- Independent review passes probed the public interface of 0.4.0 and the new code side by side on - 5,832 further cases (every variant and BCH version, invalid watermark strings, extreme strengths, - all pixel formats at 15 sizes, special floating point pixels, non-sRGB color spaces, encode and - decode), and found no difference other than the accepted one. -- `cargo clippy --workspace --all-targets -- -D warnings` and `cargo fmt --check` report nothing, - and `cargo test --workspace` passes. - -## Behavior to know about - -One case changed. With the P and Q variants, `encode` on an image with a width or height of 0 -panicked inside `remove_boundary_artifact` in 0.4.0. The new code returns -`Error::ImageProcessing(image_processing::Error::Image)` ("invalid image") at the same point, -after the watermark is validated and the model has run, so an invalid watermark is still reported -as `Error::Bits`. With the B and C variants such an image returns an empty image, as in 0.4.0. -`decode` on an empty image returns `Error::CorruptWatermark`, in 0.4.0 and in the new code. - -The following cases behave the same in 0.4.0 and in the new code; they are listed because a reader -may not expect them. - -- Thin images encode, but their watermark does not decode. On strips cut from `ripley.jpg` with a - long side of 2000 pixels, Q watermarks decoded from a short side of 96 pixels and P watermarks - from 64. A 2000 by 32 Q strip decoded to a different payload, reported as valid. A 100 by 60 - image also encodes without a decodable watermark. -- Decoders report watermarks in some unwatermarked images. The Rust P decoder reports the same - 75-bit payload for `c2pa/example.jpg` and `images/ufo_240.jpg`, which Python reads as - unwatermarked; the JavaScript decoder reports a payload for both. The Rust and C++ Q decoders - report one for `images/ghost.png` converted to RGB, and the C++ P decoder for `ripley.jpg`. -- Python and JavaScript find no watermark in `c2pa/example_wm.jpg` or - `c2pa/example_wm_signed.jpg`. The Rust B and P decoders report payloads for both, and the Rust Q - decoder reports none. -- The JavaScript example page shrinks images to 500 pixels wide before decoding. After that step it - read a different payload from one of the 40 files, a P watermark encoded by Python into - `ripley.jpg`; at full size it read the right one.