From 3a58dc6463cfd5602184834d751e5bb7aa7666ab Mon Sep 17 00:00:00 2001 From: Tom Laird-McConnell Date: Wed, 14 Oct 2026 11:09:29 -0700 Subject: [PATCH 1/2] Hold Sixel as the pixels each line shows, not as pictures (3.0.0) A Sixel is content: it paints the cells it lands on, a later Sixel paints over it, and printing over a cell replaces it. Keeping each Sixel as a picture with one placement per row, split wherever something covered part of it, left a program that repaints small regions with thousands of fragments per screen, each pinning its whole picture and each its own draw. Each line now keeps a SixelStrip: the pixels Sixel painted on it, one cell tall. A Sixel paints into the strips of the lines it covers (unset pixels under background select 1 leave what was there), printing and erasing clear those cells, and ICH/DCH, insert mode and margin scrolls move the pixels with their cells. Clones share a strip until either side writes. The image budget counts strips and drops scrollback strips first, oldest line first, never what is on screen. iTerm2 inline images are painted the same way. Kitty is unchanged. Sixel no longer appears in Placements or Images, so this is a major version: 3.0.0. README has an upgrade section. Co-Authored-By: Claude Opus 5.5 --- README.md | 102 ++++- .../Graphics/ImageAssertions.cs | 47 ++ .../Graphics/ImageCellLifetimeTests.cs | 419 ++++++++++-------- .../Graphics/SixelDecoderTests.cs | 27 +- .../Graphics/SixelPlacementTests.cs | 111 ++--- src/XTerm.NET.Tests/ITerm2OscTests.cs | 6 +- src/XTerm.NET.Tests/XtGetTcapTests.cs | 4 +- src/XTerm.NET/Buffer/BufferLine.cs | 150 ++++++- src/XTerm.NET/Graphics/SixelStrip.cs | 327 ++++++++++++++ src/XTerm.NET/InputHandler.KittyGraphics.cs | 46 +- src/XTerm.NET/Terminal.cs | 51 ++- src/XTerm.NET/XTerm.NET.csproj | 4 +- 12 files changed, 1007 insertions(+), 287 deletions(-) create mode 100644 src/XTerm.NET/Graphics/SixelStrip.cs diff --git a/README.md b/README.md index 7ed0d405..ac2c7590 100644 --- a/README.md +++ b/README.md @@ -8,11 +8,11 @@ A .NET terminal emulator library inspired by [xterm.js](https://github.com/xterm XTerm.NET provides a headless terminal emulator that parses and processes VT100/ANSI escape sequences, making it easy to host console applications in your .NET applications. -**Sixel and Kitty graphics are supported.** A picture is held as a run on the line it appears on, so -it scrolls with that line, is cleared by `ED`/`EL`, and is freed when the line falls out of the -scrollback — and a resize does nothing to it at all. Sixel is *content*, so printing over it replaces -that part of the picture; Kitty is an *overlay* ordered against the text by its z-index. See -[Images](#images). +**Sixel and Kitty graphics are supported.** Sixel is *content*: each line keeps the pixels Sixel +painted on it, a later Sixel paints over them, and printing over a cell replaces its pixels. Kitty is +an *overlay*: a picture is held as a run on the line it appears on, ordered against the text by its +z-index. Either way a picture scrolls with its line, is cleared by `ED`/`EL`, and is freed when the +line falls out of the scrollback — and a resize does nothing to it at all. See [Images](#images). ## Features @@ -31,8 +31,10 @@ that part of the picture; Kitty is an *overlay* ordered against the text by its - **Left and Right Margins** — `DECSLRM` under `DECLRMM`, so text and scrolling stay inside a pane - **Shell Integration** — `OSC 133` prompt marks survive reflow, `OSC 8` hyperlinks, `OSC 52` clipboard, and desktop notifications (`OSC 9`/`99`/`777`) -- **Sixel Graphics** — Decodes Sixel images (`ESC P … q`) and holds them as runs on the lines they - appear on, so `img2sixel`, `chafa`, `lsix` and `timg` work against a host that renders them +- **Sixel Graphics** — Decodes Sixel images (`ESC P … q`) and paints them into the lines they land + on, so `img2sixel`, `chafa`, `lsix` and `timg` work against a host that renders them — and a + program repainting small regions over and over costs one strip of pixels per line, however many + Sixels it sends - **Kitty Graphics** — Decodes the Kitty protocol (`ESC _ G …`), including chunked transmission, PNG, transmit-once/place-many by image id, animation, and U+10EEEE Unicode placeholders, so `icat`, `chafa -f kitty`, `timg -pk`, `yazi` and `image.nvim` work the same way @@ -45,6 +47,53 @@ that part of the picture; Kitty is an *overlay* ordered against the text by its ## Compliance ![Compliance](https://raw.githubusercontent.com/tomlm/Iciclecreek.Avalonia.Terminal/main/compliance.png) +## Upgrading to 3.0 + +**3.0 changes how Sixel is stored.** If you only host a terminal and draw text, or draw only Kitty +pictures, nothing changes. If you draw Sixel, you draw it from a new place. + +In 2.x a Sixel was a picture: a `TerminalImage` and one `LinePlacement` per row, split wherever +printing or a later Sixel covered part of it. A program that repaints small regions — a cursor, a +hover highlight, a diffing framebuffer such as SixelBuffer — left a patchwork behind: thousands of +fragments per screen, each still pinning its whole picture and each its own draw. Sixel is content, +so in 3.0 a line keeps the *pixels* Sixel painted on it, in a `SixelStrip`, and the picture that +painted them is not kept at all. + +**What changed.** + +| 2.x | 3.0 | +| --- | --- | +| A Sixel appears in `line.Placements` and `line.Images` | It is painted into `line.Sixel`; `Placements` and `Images` hold Kitty pictures only | +| `PlacementKind.Sixel` runs | Never created. The value remains so code that compares against it still compiles | +| `TryGetPlacementAt` / `TryGetImageAt` find Sixel | They find Kitty pictures; ask `line.Sixel?.HasPixelsAt(column)` | +| A later Sixel splits the one under it | It paints over it: opaque pixels replace, unset pixels (background select 1) leave what was there | +| `ICH`/`DCH`, insert mode and margin scrolls left Sixel behind | Its pixels move with their cells | +| Kitty deletes by position or z-index also removed Sixel runs | They no longer touch Sixel; `a=d,d=a` still clears everything | +| `MaxImageBytes` counted pictures | It counts pictures and strips; scrollback strips go first, oldest line first, and what is on screen is never taken | +| iTerm2 inline images were Sixel placements | They are painted into strips, as Sixel is | + +`line.HasImages` is true for a line with a strip as well as one with Kitty runs. + +**What a renderer does now.** Per line: Kitty pictures behind the text (negative `ZIndex`), then the +text, then the line's Sixel strip, then Kitty pictures in front. The strip is one blit: + +```csharp +if (line.Sixel is { } strip) +{ + // BGRA8888, straight alpha, top row first: PaintedColumns cells of CellWidth x CellHeight pixels. + // Cache the upload against the strip and upload again when strip.Version changes. + var cols = Math.Min(strip.PaintedColumns, line.Length); + DrawImage(bitmapFor(strip), + source: (0, 0, cols * strip.CellWidth, strip.CellHeight), + dest: (0, row * cellHeight, cols * cellWidth, cellHeight)); +} +``` + +A strip is painted at the cell size in force when its Sixels arrived, so after a font change it is +stretched onto the new grid, as a picture is. A strip can be painted by the thread writing to the +terminal while you read it: read `Version` before copying and again after, and treat a copy whose +version moved as stale. + ## Upgrading to 2.0 **2.0 targets .NET 10** and removes the per-cell image API. If you only host a terminal and draw @@ -434,15 +483,21 @@ are reported honestly either way. ### Images Two graphics protocols are decoded — Sixel (`ESC P … q … ESC \`) and Kitty -(`ESC _ G … ESC \`) — and both end up in the same place: a **run held by the line** the picture -appears on. Nothing about a picture is written into cells, which is what makes it survive a resize: -a run keeps its natural width, so narrowing a window shows less of a picture instead of destroying -it. The run scrolls with its line, is cleared by `ED`/`EL`, and is freed when the line falls out of -the scrollback. - -A **placement** is one appearance of a picture: which image it draws, which rectangle of that image -it takes, and how many cells it fills. Sixel makes one placement per image. Kitty can transmit a -picture once and place it many times, so several placements may share one `TerminalImage`. +(`ESC _ G … ESC \`) — and both end up held by the **line** the picture appears on. Nothing about a +picture is written into cells, which is what makes it survive a resize: what a line holds keeps its +natural width, so narrowing a window shows less of a picture instead of destroying it. It scrolls +with its line, is cleared by `ED`/`EL`, and is freed when the line falls out of the scrollback. + +The two protocols differ in what a picture *is*: + +- **Sixel is content.** Each line keeps the pixels Sixel painted on it, in a `SixelStrip` + (`line.Sixel`): one cell tall, as wide as anything painted. A later Sixel paints over it, printing + or erasing over a cell clears that cell's pixels, and an insert or delete moves them with their + cells. The picture that painted them is not kept. +- **Kitty is an overlay.** A **placement** is one appearance of a picture: which image it draws, + which rectangle of that image it takes, and how many cells it fills. It is held as one run per line + in `line.Placements`. Kitty can transmit a picture once and place it many times, so several + placements may share one `TerminalImage`. **Tell the terminal your cell size.** XTerm.NET is headless and cannot measure a font, so it cannot work out how many columns an image covers unless you say. Set these from your renderer's metrics, @@ -477,7 +532,11 @@ up to a whole row of the control's height belongs to no row at all. An applicati figure by the row count is told the terminal is taller than it is, sizes a picture to fill it, and the surplus runs off the bottom and scrolls the screen. -**Drawing the runs.** This is per *line*, not per cell — a run is one blit, not a row of tiles: +**Drawing a line's Sixel pixels.** One blit per line, after the text and before Kitty pictures in +front of it — see [Upgrading to 3.0](#upgrading-to-30) for the code. `strip.PaintedColumns` is how far +anything is painted, so a narrow picture on a wide line does not draw the rest of the row. + +**Drawing the Kitty runs.** This is per *line*, not per cell — a run is one blit, not a row of tiles: ```csharp var line = buffer.Lines[buffer.YDisp + row]; @@ -535,12 +594,13 @@ ordinary case of one picture at a time — a renderer handling overlapping pictu | `KittyGraphicsEnabled` | `true` | Decode Kitty graphics sequences and answer their queries | | `CellWidthPixels` / `CellHeightPixels` | `10` / `20` | Cell size images are laid out against | | `MaxSixelPixels` | `4_000_000` | Largest single image accepted | -| `MaxImageBytes` | `64 MB` | Budget for image data live in the buffer; oldest are dropped past it | +| `MaxImageBytes` | `64 MB` | Budget for image data live in the buffer — Kitty pictures and Sixel strips; past it, scrollback strips go first, then the oldest pictures | | `MaxImageRegistryBytes` | `32 MB` | Budget for transmitted-but-unplaced Kitty images, evicted oldest first | -Images are dropped when the terminal is resized to a different **column** count, because reflow -re-wraps lines by copying ranges of cells and the pieces would reassemble in the wrong places. A -change of row count alone keeps them. +A change of column count keeps pictures: narrowing shows less of them and widening shows more. Only +lines that are part of a wrap chain drop theirs, because reflow re-wraps those by copying ranges of +cells between lines and the pieces would reassemble in the wrong places. A change of row count alone +keeps everything. #### Kitty graphics diff --git a/src/XTerm.NET.Tests/Graphics/ImageAssertions.cs b/src/XTerm.NET.Tests/Graphics/ImageAssertions.cs index 8f1221cd..3789bb11 100644 --- a/src/XTerm.NET.Tests/Graphics/ImageAssertions.cs +++ b/src/XTerm.NET.Tests/Graphics/ImageAssertions.cs @@ -94,6 +94,53 @@ public static (int Col, int Row)? TileAt(Terminal terminal, int col, int screenR return (source.X / image.CellWidth, source.Y / image.CellHeight); } + /// The Sixel pixels on a screen row, or null when no Sixel has painted it. + public static SixelStrip? StripOn(Terminal terminal, int screenRow) + => terminal.Buffer.Lines[terminal.Buffer.YBase + screenRow]?.Sixel; + + /// + /// Whether a screen position shows Sixel pixels -- the question asks of a + /// Kitty run, asked of the pixels Sixel paints instead. + /// + public static bool IsSixelAt(Terminal terminal, int col, int screenRow) + { + var line = terminal.Buffer.Lines[terminal.Buffer.YBase + screenRow]; + return line?.Sixel is { } strip && col < line.Length && strip.HasPixelsAt(col); + } + + /// + /// The Sixel pixel at (, ) inside a cell, as RGBA, or null + /// when the cell shows none. + /// + public static (byte R, byte G, byte B, byte A)? SixelPixel(Terminal terminal, int col, int screenRow, int x = 0, int y = 0) + { + if (!IsSixelAt(terminal, col, screenRow)) + return null; + + var strip = StripOn(terminal, screenRow)!; + var offset = y * strip.Stride + (col * strip.CellWidth + x) * TerminalImage.BytesPerPixel; + var p = strip.Pixels.Span; + return (p[offset + 2], p[offset + 1], p[offset], p[offset + 3]); + } + + /// How many cells of the buffer show Sixel pixels, clipped to each line's width. + public static int VisibleSixelCells(Terminal terminal) + { + var count = 0; + for (var i = 0; i < terminal.Buffer.Lines.Length; i++) + { + var line = terminal.Buffer.Lines[i]; + if (line?.Sixel is not { } strip) + continue; + + for (var c = 0; c < System.Math.Min(strip.Columns, line.Length); c++) + if (strip.HasPixelsAt(c)) + count++; + } + + return count; + } + /// Every run on a screen row, in the order they were placed. public static IReadOnlyList PlacementsOn(Terminal terminal, int screenRow) { diff --git a/src/XTerm.NET.Tests/Graphics/ImageCellLifetimeTests.cs b/src/XTerm.NET.Tests/Graphics/ImageCellLifetimeTests.cs index 92633573..dbdc37da 100644 --- a/src/XTerm.NET.Tests/Graphics/ImageCellLifetimeTests.cs +++ b/src/XTerm.NET.Tests/Graphics/ImageCellLifetimeTests.cs @@ -6,22 +6,35 @@ namespace XTerm.Tests.Graphics; /// -/// The behaviour that justifies keeping images on the cells rather than in an overlay: a picture -/// is terminal content, and everything that happens to text has to happen to it. +/// The behaviour that makes Sixel terminal content: everything that happens to text happens to the +/// pixels a Sixel painted. /// -/// Printing over a cell replaces it, erasing clears it, scrolling carries it, and falling -/// off the end of the scrollback disposes of it -- none of which needed code, because a cell is a -/// struct and every one of those paths already builds or copies whole cells. These tests exist to -/// keep it that way. +/// Each line keeps the pixels Sixel painted on it (), not the pictures +/// that painted them. Printing over a cell replaces its pixels, erasing clears them, a later Sixel +/// paints over them, scrolling carries them with their line, and falling off the end of the +/// scrollback disposes of them. These tests exist to keep it that way. /// public class ImageCellLifetimeTests { private const string Esc = "\u001b"; private const string St = Esc + "\\"; - /// Four pixels wide, twelve tall: two cells across, four down. + /// Four pixels wide, twelve tall, red: two cells across, four down. private const string TwoByFourCells = "#0;2;100;0;0!4~-!4~"; + /// The same size in blue. + private const string BlueTwoByFour = "#0;2;0;0;100!4~-!4~"; + + /// One cell across, four down, green. + private const string GreenOneByFour = "#0;2;0;100;0!2~-!2~"; + + /// Two by four, a different colour in each cell row: red, green, blue, yellow. + private const string Striped = "#0;2;100;0;0!4F$#1;2;0;100;0!4w-#2;2;0;0;100!4F$#3;2;100;100;0!4w"; + + private static readonly (byte R, byte G, byte B, byte A) Red = (255, 0, 0, 255); + private static readonly (byte R, byte G, byte B, byte A) Green = (0, 255, 0, 255); + private static readonly (byte R, byte G, byte B, byte A) Blue = (0, 0, 255, 255); + private static Terminal Fresh(Action? configure = null) { var options = new TerminalOptions @@ -41,41 +54,26 @@ private static void WriteSixel(Terminal terminal, string body = TwoByFourCells) private static BufferCell Cell(Terminal terminal, int col, int screenRow) => terminal.Buffer.Lines[terminal.Buffer.YBase + screenRow]![col]; - private static int ImageCellCount(Terminal terminal) - { - int count = 0; - for (int i = 0; i < terminal.Buffer.Lines.Length; i++) - { - var line = terminal.Buffer.Lines[i]; - if (line is null) - continue; - // Counted from the runs and clamped to the line's width, so this reports what is - // VISIBLE — anything wider is hidden by the window, not destroyed. - foreach (var placement in line.Placements) - { - var end = Math.Min(placement.EndColumn, line.Length); - count += Math.Max(0, end - placement.Column); - } - } - return count; - } + private static BufferLine Line(Terminal terminal, int screenRow) + => terminal.Buffer.Lines[terminal.Buffer.YBase + screenRow]!; + + private static int SixelCells(Terminal terminal) => ImageAssertions.VisibleSixelCells(terminal); [Fact] - public void Sixel_over_a_sixel_replaces_the_runs_it_covers() + public void Sixel_over_a_sixel_paints_over_it() { var terminal = Fresh(); WriteSixel(terminal); - var first = ImageAssertions.ImageAt(terminal, 0, 0); - // The same picture drawn again at the same place: one run per row, not two terminal.Write($"{Esc}[1;1H"); - WriteSixel(terminal); - - var line = terminal.Buffer.Lines[terminal.Buffer.YBase]!; - Assert.Single(line.Placements); - Assert.Single(line.Images); - Assert.NotSame(first, ImageAssertions.ImageAt(terminal, 0, 0)); - Assert.Equal(8, ImageCellCount(terminal)); + WriteSixel(terminal, BlueTwoByFour); + + // One strip per line holding the newer pixels; nothing of the older picture is kept, and + // nothing about either becomes a run. + Assert.Equal(Blue, ImageAssertions.SixelPixel(terminal, 0, 0)); + Assert.Equal(Blue, ImageAssertions.SixelPixel(terminal, 1, 3)); + Assert.Empty(Line(terminal, 0).Placements); + Assert.Equal(8, SixelCells(terminal)); } [Fact] @@ -83,37 +81,60 @@ public void Sixel_over_part_of_a_sixel_keeps_the_rest_of_it() { var terminal = Fresh(); WriteSixel(terminal); - var first = ImageAssertions.ImageAt(terminal, 0, 0); // A one-cell-wide picture on the second column, over the right half of the first one terminal.Write($"{Esc}[1;2H"); - WriteSixel(terminal, "#0;2;0;100;0!2~-!2~"); + WriteSixel(terminal, GreenOneByFour); - var line = terminal.Buffer.Lines[terminal.Buffer.YBase]!; - Assert.Equal(2, line.Placements.Count); - Assert.Same(first, ImageAssertions.ImageAt(terminal, 0, 0)); - Assert.NotSame(first, ImageAssertions.ImageAt(terminal, 1, 0)); - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(Red, ImageAssertions.SixelPixel(terminal, 0, 0)); + Assert.Equal(Green, ImageAssertions.SixelPixel(terminal, 1, 0)); + Assert.Equal(8, SixelCells(terminal)); } + /// + /// The case the strips exist for: a program repainting a small region over and over keeps one + /// strip per line, the same size it started, however many Sixels it sends. + /// [Fact] - public void Repainting_a_cell_many_times_does_not_accumulate_runs() + public void Repainting_a_cell_many_times_accumulates_nothing() { var terminal = Fresh(); WriteSixel(terminal); + var bytes = Enumerable.Range(0, 4).Sum(row => ImageAssertions.StripOn(terminal, row)!.ByteCount); for (int i = 0; i < 50; i++) { terminal.Write($"{Esc}[1;2H"); - WriteSixel(terminal, "#0;2;0;0;100!2~-!2~"); + WriteSixel(terminal, i % 2 == 0 ? GreenOneByFour : "#0;2;0;0;100!2~-!2~"); } for (int row = 0; row < 4; row++) { - var line = terminal.Buffer.Lines[terminal.Buffer.YBase + row]!; - Assert.Equal(2, line.Placements.Count); - Assert.Equal(2, line.Images.Count); + Assert.Empty(Line(terminal, row).Placements); + Assert.Equal(Red, ImageAssertions.SixelPixel(terminal, 0, row)); + Assert.Equal(Blue, ImageAssertions.SixelPixel(terminal, 1, row)); } + + Assert.Equal(bytes, Enumerable.Range(0, 4).Sum(row => ImageAssertions.StripOn(terminal, row)!.ByteCount)); + } + + /// + /// Background select 1: a Sixel's unset pixels are transparent, and whatever was painted under + /// them shows through -- which a program relies on to draw a sprite over a picture. + /// + [Fact] + public void A_transparent_sixel_leaves_what_is_under_its_unset_pixels() + { + var terminal = Fresh(); + WriteSixel(terminal); + + // Only the top pixel row of each sixel is set ('@'): rows 1 and 2 of the first cell stay red. + terminal.Write($"{Esc}[1;1H"); + WriteSixel(terminal, "#0;2;0;100;0!4@"); + + Assert.Equal(Green, ImageAssertions.SixelPixel(terminal, 0, 0, y: 0)); + Assert.Equal(Red, ImageAssertions.SixelPixel(terminal, 0, 0, y: 1)); + Assert.Equal(Red, ImageAssertions.SixelPixel(terminal, 0, 0, y: 2)); } [Fact] @@ -124,12 +145,12 @@ public void Printing_over_a_tile_replaces_it_with_the_character() terminal.Write($"{Esc}[1;1HX"); - Assert.Null(ImageAssertions.ImageAt(terminal, 0, 0)); + Assert.False(ImageAssertions.IsSixelAt(terminal, 0, 0)); Assert.Equal("X", Cell(terminal, 0, 0).Content); // and only that cell - Assert.NotNull(ImageAssertions.ImageAt(terminal, 1, 0)); - Assert.NotNull(ImageAssertions.ImageAt(terminal, 0, 1)); + Assert.True(ImageAssertions.IsSixelAt(terminal, 1, 0)); + Assert.True(ImageAssertions.IsSixelAt(terminal, 0, 1)); } [Fact] @@ -140,9 +161,12 @@ public void Erase_in_line_clears_the_tiles_on_that_row() terminal.Write($"{Esc}[1;1H{Esc}[K"); - Assert.Null(ImageAssertions.ImageAt(terminal, 0, 0)); - Assert.Null(ImageAssertions.ImageAt(terminal, 1, 0)); - Assert.NotNull(ImageAssertions.ImageAt(terminal, 0, 1)); + Assert.False(ImageAssertions.IsSixelAt(terminal, 0, 0)); + Assert.False(ImageAssertions.IsSixelAt(terminal, 1, 0)); + Assert.True(ImageAssertions.IsSixelAt(terminal, 0, 1)); + + // A line with nothing left drops its strip rather than keeping a blank one. + Assert.Null(ImageAssertions.StripOn(terminal, 0)); } [Fact] @@ -153,7 +177,7 @@ public void Erase_in_display_clears_every_tile_on_screen() terminal.Write($"{Esc}[2J"); - Assert.Equal(0, ImageCellCount(terminal)); + Assert.Equal(0, SixelCells(terminal)); } [Fact] @@ -164,8 +188,43 @@ public void Erase_characters_clears_the_tiles_it_covers() terminal.Write($"{Esc}[1;1H{Esc}[1X"); - Assert.Null(ImageAssertions.ImageAt(terminal, 0, 0)); - Assert.NotNull(ImageAssertions.ImageAt(terminal, 1, 0)); + Assert.False(ImageAssertions.IsSixelAt(terminal, 0, 0)); + Assert.True(ImageAssertions.IsSixelAt(terminal, 1, 0)); + } + + /// + /// An insert moves the text after the cursor right, and the picture it sits in moves with it. + /// Pictures held as runs stayed where they were while their cells moved out from under them. + /// + [Fact] + public void Inserting_characters_moves_the_pixels_with_their_cells() + { + var terminal = Fresh(); + WriteSixel(terminal, Striped); + + terminal.Write($"{Esc}[1;1H{Esc}[3@"); // ICH 3 on the top row + + Assert.False(ImageAssertions.IsSixelAt(terminal, 0, 0)); + Assert.False(ImageAssertions.IsSixelAt(terminal, 2, 0)); + Assert.Equal(Red, ImageAssertions.SixelPixel(terminal, 3, 0)); + Assert.Equal(Red, ImageAssertions.SixelPixel(terminal, 4, 0)); + + // Only that row: the one below still starts at the left margin. + Assert.Equal(Green, ImageAssertions.SixelPixel(terminal, 0, 1)); + } + + [Fact] + public void Deleting_characters_moves_the_pixels_with_their_cells() + { + var terminal = Fresh(); + terminal.Write($"{Esc}[1;4H"); + WriteSixel(terminal, Striped); + + terminal.Write($"{Esc}[1;1H{Esc}[2P"); // DCH 2 on the top row + + Assert.Equal(Red, ImageAssertions.SixelPixel(terminal, 1, 0)); + Assert.Equal(Red, ImageAssertions.SixelPixel(terminal, 2, 0)); + Assert.False(ImageAssertions.IsSixelAt(terminal, 3, 0)); } [Fact] @@ -176,99 +235,88 @@ public void A_full_reset_clears_every_tile() terminal.Write($"{Esc}c"); - Assert.Equal(0, ImageCellCount(terminal)); + Assert.Equal(0, SixelCells(terminal)); } [Fact] public void Scrolling_carries_the_tiles_with_their_lines() { var terminal = Fresh(); - WriteSixel(terminal); - var image = ImageAssertions.ImageAt(terminal, 0, 0); - Assert.NotNull(image); + WriteSixel(terminal, Striped); // The absolute line the top of the image went onto. Scrolling moves the viewport over the // buffer, so this index is what should stay put while the screen row changes. var topLine = terminal.Buffer.YBase; + var strip = terminal.Buffer.Lines[topLine]!.Sixel; + Assert.NotNull(strip); // Push the screen up by two rows from the bottom. terminal.Write($"{Esc}[10;1H\r\n\r\n"); Assert.Equal(2, terminal.Buffer.YBase - topLine); - // Nothing copied the tiles anywhere: the same line object still holds them, unchanged. - var moved = terminal.Buffer.Lines[topLine]!; - Assert.True(moved.TryGetImageAt(0, out var movedImage) && ReferenceEquals(movedImage, image), - "the run did not travel with its line"); - Assert.True(moved.TryGetPlacementAt(0, out var movedPlacement)); - Assert.Equal(0, movedPlacement.SrcY); + // Nothing copied the pixels anywhere: the same line object still holds the same strip. + Assert.Same(strip, terminal.Buffer.Lines[topLine]!.Sixel); // Which puts the top of the picture two rows higher on screen than it was. - Assert.True(ReferenceEquals(ImageAssertions.ImageAt(terminal, 0, -2), image)); - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(Red, ImageAssertions.SixelPixel(terminal, 0, -2)); + Assert.Equal(Green, ImageAssertions.SixelPixel(terminal, 0, -1)); + Assert.Equal(8, SixelCells(terminal)); } /// - /// The disposal story: an image dies with the last cell holding it, so a picture that scrolls - /// out of a short scrollback leaves nothing behind to evict. + /// The disposal story: pixels die with the line holding them, so a picture that scrolls out of a + /// short scrollback leaves nothing behind to evict. /// [Fact] public void An_image_scrolled_out_of_the_scrollback_leaves_no_references() { var terminal = Fresh(o => o.Scrollback = 4); WriteSixel(terminal); - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(8, SixelCells(terminal)); terminal.Write($"{Esc}[10;1H"); for (int i = 0; i < 40; i++) terminal.Write("\r\n"); - Assert.Equal(0, ImageCellCount(terminal)); + Assert.Equal(0, SixelCells(terminal)); } /// - /// A change of width keeps the pictures. + /// A change of width keeps the pictures: a strip keeps its natural width and the renderer draws + /// as much of it as the line allows. Only a wrap chain drops them, because reflow moves cells + /// between lines. /// - /// - /// This test used to assert the opposite, and the reasoning it carried — reflow re-wraps a - /// logical line by copying ranges of cells, so tiles carried through would reassemble as a - /// shuffled mosaic — was right about reflow and wrong about the blast radius. It applied to lines - /// that actually re-wrap, and the code dropped every picture in the buffer on any width change at - /// all, including widening a window, which is the most common resize there is. - /// With a picture held as a run on its line rather than tiles in cells, there is nothing to - /// shuffle: the renderer draws as much of the run as the width allows. Narrowing shows less, - /// widening shows more, and the wrap-chain case is still dropped, on its own, below. - /// [Fact] public void A_change_of_width_keeps_the_images() { var terminal = Fresh(); WriteSixel(terminal); - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(8, SixelCells(terminal)); terminal.Resize(15, 10); - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(8, SixelCells(terminal)); terminal.Resize(40, 10); - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(8, SixelCells(terminal)); } /// /// Narrowing past a picture hides the overhang rather than destroying it, and widening brings it - /// back — because a run keeps its natural width and only the drawing is clipped. + /// back. /// [Fact] public void Narrowing_past_a_picture_hides_it_and_widening_restores_it() { var terminal = Fresh(); WriteSixel(terminal); - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(8, SixelCells(terminal)); terminal.Resize(1, 10); // the picture is two columns wide - Assert.Equal(4, ImageCellCount(terminal)); + Assert.Equal(4, SixelCells(terminal)); terminal.Resize(20, 10); - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(8, SixelCells(terminal)); } /// A change of height moves whole lines, so there is nothing to be confused about. @@ -280,7 +328,7 @@ public void A_change_of_height_alone_keeps_the_images() terminal.Resize(20, 14); - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(8, SixelCells(terminal)); } [Fact] @@ -297,26 +345,31 @@ public void Clearing_a_dropped_image_leaves_a_blank_rather_than_a_hole() } /// - /// The backstop for the case reference counting cannot reach: a deep scrollback full of - /// pictures, every one still referenced and every one still in memory. + /// The backstop for a deep scrollback full of pictures: over budget, the Sixel pixels furthest + /// back in the history go first. /// [Fact] - public void The_oldest_images_are_dropped_when_the_budget_is_exceeded() + public void Scrollback_pixels_are_dropped_first_when_the_budget_is_exceeded() { - // Each image here is 4x12 BGRA -- 192 bytes -- so a 200 byte budget holds exactly one. - var terminal = Fresh(o => o.MaxImageBytes = 200); - + // Each painted line is a 20-cell strip of 2x3 BGRA cells: 480 bytes. Two pictures of four + // lines are 3840; 2500 holds one of them. + var terminal = Fresh(o => o.MaxImageBytes = 2500); WriteSixel(terminal); - var first = ImageAssertions.ImageAt(terminal, 0, 0); - Assert.NotNull(first); + var oldest = terminal.Buffer.Lines[terminal.Buffer.YBase]!; + // Scroll the first picture off the screen into the scrollback, then draw a second. + terminal.Write($"{Esc}[10;1H" + string.Concat(Enumerable.Repeat("\r\n", 10))); + terminal.Write($"{Esc}[1;1H"); WriteSixel(terminal); - var second = ImageAssertions.ImageAt(terminal, 0, 4); - Assert.NotNull(second); - Assert.Null(ImageAssertions.ImageAt(terminal, 0, 0)); - Assert.True(ReferenceEquals(ImageAssertions.ImageAt(terminal, 0, 4), second), - "the newest image should be the one that survives"); + terminal.EnforceImageBudget(); + + // Just enough goes, oldest line first: three of the old picture's four lines take the + // buffer under budget, so the fourth -- the newest of them -- stays. + Assert.Null(oldest.Sixel); + Assert.Equal(8 + 2, SixelCells(terminal)); + for (int row = 0; row < 4; row++) + Assert.True(ImageAssertions.IsSixelAt(terminal, 0, row), $"the picture on screen should survive (row {row})"); } [Fact] @@ -326,80 +379,53 @@ public void A_generous_budget_keeps_both_images() WriteSixel(terminal); WriteSixel(terminal); + terminal.EnforceImageBudget(); - Assert.NotNull(ImageAssertions.ImageAt(terminal, 0, 0)); - Assert.NotNull(ImageAssertions.ImageAt(terminal, 0, 4)); + Assert.True(ImageAssertions.IsSixelAt(terminal, 0, 0)); + Assert.True(ImageAssertions.IsSixelAt(terminal, 0, 4)); } /// - /// Evicting one picture leaves another that shares its line. + /// What is on screen is never taken: at most one screen of pixels, and it is what the user is + /// looking at. /// - /// - /// Caught in review. Dropping the whole line because one of its pictures was over budget took - /// the others with it — more destructive than the per-cell code this replaced, and it evicted - /// images the sweep had just decided to keep. - /// [Fact] - public void Evicting_one_image_leaves_another_on_the_same_line() + public void The_budget_never_takes_what_is_on_screen() { - // Two 192-byte images; a 400 byte budget holds both, then a third forces one out. - var terminal = Fresh(o => o.MaxImageBytes = 400); + var terminal = Fresh(o => o.MaxImageBytes = 1); WriteSixel(terminal); - terminal.Write($"{Esc}[1;5H"); - WriteSixel(terminal); - - var line = terminal.Buffer.Lines[terminal.Buffer.YBase]!; - Assert.Equal(2, line.Images.Count); - var newer = line.Images[1]; - - // A third image pushes past the budget, dooming the oldest — which shares this line. - terminal.Write($"{Esc}[10;1H"); WriteSixel(terminal); + terminal.EnforceImageBudget(); - Assert.True(ReferenceEquals(newer, line.Images.SingleOrDefault()), - "the line's other picture should survive its neighbour being evicted"); + Assert.Equal(16, SixelCells(terminal)); } /// - /// Printing over one picture releases it, while another on the same line stays. + /// Printing over one picture clears its pixels and leaves another on the same line alone. /// - /// - /// Caught in review. Ownership is derived from the runs, so anything that removes one has to - /// rebuild it — waiting for the run list to empty kept a picture alive that nothing displayed, - /// and hid it from the budget sweep, which decides what is live by walking runs. - /// [Fact] - public void Overwriting_one_picture_releases_only_that_one() + public void Overwriting_one_picture_leaves_another_on_the_same_line() { var terminal = Fresh(); WriteSixel(terminal); terminal.Write($"{Esc}[1;5H"); - WriteSixel(terminal); - - var line = terminal.Buffer.Lines[terminal.Buffer.YBase]!; - Assert.Equal(2, line.Images.Count); - var survivor = line.Images[1]; + WriteSixel(terminal, BlueTwoByFour); // Print across the whole span of the first picture, which covers columns 0..1. terminal.Write($"{Esc}[1;1HXX"); - Assert.True(ReferenceEquals(survivor, line.Images.SingleOrDefault()), - "the overwritten picture should be released and the other kept"); + Assert.False(ImageAssertions.IsSixelAt(terminal, 0, 0)); + Assert.False(ImageAssertions.IsSixelAt(terminal, 1, 0)); + Assert.Equal(Blue, ImageAssertions.SixelPixel(terminal, 4, 0)); + Assert.Equal(Blue, ImageAssertions.SixelPixel(terminal, 5, 0)); } /// - /// A cell under a picture is an ordinary space, and that is now correct rather than a bug. + /// A cell under a picture is an ordinary space: nothing about a picture is drawn from cells, so + /// cells beneath one coalesce exactly like the spaces they are. /// - /// - /// This inverts what these tests used to assert. When a cell carried an image reference and a - /// tile coordinate, two cells showing different pieces of a picture HAD to compare unequal, - /// because renderers coalesce adjacent cells into one run by comparing them and merging two - /// different tiles would have drawn the wrong thing. With pictures held as runs on the line, - /// nothing about a picture is drawn from cells — so cells beneath one should coalesce exactly - /// like the spaces they are, and the distinction has no work left to do. - /// [Fact] public void A_cell_under_a_picture_is_an_ordinary_space() { @@ -409,25 +435,20 @@ public void A_cell_under_a_picture_is_an_ordinary_space() Assert.Equal(BufferCell.Space, Cell(terminal, 0, 0)); Assert.Equal(Cell(terminal, 0, 0), Cell(terminal, 1, 0)); - // And the picture is still there — held by the line, which is the whole point. - Assert.NotNull(ImageAssertions.ImageAt(terminal, 0, 0)); + // And the picture is still there -- held by the line, which is the whole point. + Assert.True(ImageAssertions.IsSixelAt(terminal, 0, 0)); } - /// - /// What must stay distinguishable is the RUNS: each line shows its own slice of the picture. - /// + /// Each line shows its own band of the picture. [Fact] - public void Runs_on_different_lines_carry_different_slices() + public void Lines_carry_different_bands_of_the_picture() { var terminal = Fresh(); - WriteSixel(terminal); + WriteSixel(terminal, Striped); - var first = ImageAssertions.PlacementAt(terminal, 0, 0); - var second = ImageAssertions.PlacementAt(terminal, 0, 1); - - Assert.NotNull(first); - Assert.NotNull(second); - Assert.NotEqual(first!.Value.SrcY, second!.Value.SrcY); + Assert.Equal(Red, ImageAssertions.SixelPixel(terminal, 0, 0)); + Assert.Equal(Green, ImageAssertions.SixelPixel(terminal, 0, 1)); + Assert.Equal(Blue, ImageAssertions.SixelPixel(terminal, 0, 2)); } [Fact] @@ -437,10 +458,10 @@ public void The_alternate_buffer_keeps_its_own_images() WriteSixel(terminal); terminal.Write($"{Esc}[?1049h"); // to the alternate screen - Assert.Equal(0, ImageCellCount(terminal)); + Assert.Equal(0, SixelCells(terminal)); terminal.Write($"{Esc}[?1049l"); // and back - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(8, SixelCells(terminal)); } /// @@ -449,45 +470,38 @@ public void The_alternate_buffer_keeps_its_own_images() /// /// /// DECCARA sets attributes over an area and never touches a character, so a picture inside that - /// area is not being overwritten and must not be split. Going through the ordinary text-write - /// path made it look like one -- that path splits the placement at the column being written, - /// which is right for printing over a picture and wrong for recolouring the cell under it. + /// area is not being overwritten and must not be cleared. /// [Fact] public void An_attribute_change_over_a_picture_leaves_it_whole() { var terminal = Fresh(); WriteSixel(terminal); - var image = ImageAssertions.ImageAt(terminal, 0, 0); - Assert.NotNull(image); - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(8, SixelCells(terminal)); // Bold over the whole picture and then some. terminal.Write($"{Esc}[2*x{Esc}[1;1;4;4;1$r"); - Assert.Equal(8, ImageCellCount(terminal)); - Assert.True(ReferenceEquals(ImageAssertions.ImageAt(terminal, 0, 0), image)); - Assert.True(ReferenceEquals(ImageAssertions.ImageAt(terminal, 1, 3), image)); + Assert.Equal(8, SixelCells(terminal)); + Assert.Equal(Red, ImageAssertions.SixelPixel(terminal, 1, 3)); Assert.True(terminal.Buffer.Lines[terminal.Buffer.YBase]![0].Attributes.IsBold()); } /// /// And a request that changes nothing must not even write the cells back, because writing a - /// cell back unchanged is still a write as far as the placements are concerned. + /// cell back unchanged is still a write as far as the pixels are concerned. /// [Fact] public void An_attribute_change_naming_nothing_we_implement_touches_no_cell() { var terminal = Fresh(); WriteSixel(terminal); - var image = ImageAssertions.ImageAt(terminal, 0, 0); - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(8, SixelCells(terminal)); // 31 is a colour, which DECCARA does not carry; the request names nothing this implements. terminal.Write($"{Esc}[2*x{Esc}[1;1;4;4;31$r"); - Assert.Equal(8, ImageCellCount(terminal)); - Assert.True(ReferenceEquals(ImageAssertions.ImageAt(terminal, 0, 0), image)); + Assert.Equal(8, SixelCells(terminal)); } /// A DECRARA whose toggles cancel each other is the same nothing. @@ -496,11 +510,56 @@ public void Toggles_that_cancel_leave_the_picture_and_the_rendition_alone() { var terminal = Fresh(); WriteSixel(terminal); - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(8, SixelCells(terminal)); terminal.Write($"{Esc}[2*x{Esc}[1;1;4;4;1;1$t"); - Assert.Equal(8, ImageCellCount(terminal)); + Assert.Equal(8, SixelCells(terminal)); Assert.False(terminal.Buffer.Lines[terminal.Buffer.YBase]![0].Attributes.IsBold()); } + + /// + /// A copy of a line shares its pixels until one of them changes, and then each keeps its own: + /// a frame captured for a renderer must not see the live line being painted. + /// + [Fact] + public void A_copy_of_a_line_keeps_its_pixels_when_the_original_is_painted() + { + var terminal = Fresh(); + WriteSixel(terminal); + var line = Line(terminal, 0); + + var clone = line.Clone(); + Assert.Same(line.Sixel, clone.Sixel); + + terminal.Write($"{Esc}[1;1H"); + WriteSixel(terminal, BlueTwoByFour); + + Assert.NotSame(line.Sixel, clone.Sixel); + var pixels = clone.Sixel!.Pixels.Span; + Assert.Equal((byte)255, pixels[2]); // still red: B, G, R, A + Assert.Equal((byte)0, pixels[0]); + } + + /// + /// A Sixel laid out against a new cell size rescales what the line already shows, so every pixel + /// in a strip agrees on how big a cell is. + /// + [Fact] + public void A_sixel_at_a_new_cell_size_rescales_the_strip() + { + var terminal = Fresh(); + WriteSixel(terminal); + + terminal.Options.CellWidthPixels = 4; + terminal.Options.CellHeightPixels = 6; + terminal.Write($"{Esc}[1;5H"); + WriteSixel(terminal, "#0;2;0;0;100!4~"); + + var strip = ImageAssertions.StripOn(terminal, 0)!; + Assert.Equal(4, strip.CellWidth); + Assert.Equal(6, strip.CellHeight); + Assert.Equal(Red, ImageAssertions.SixelPixel(terminal, 0, 0, x: 3, y: 5)); + Assert.Equal(Blue, ImageAssertions.SixelPixel(terminal, 4, 0)); + } } diff --git a/src/XTerm.NET.Tests/Graphics/SixelDecoderTests.cs b/src/XTerm.NET.Tests/Graphics/SixelDecoderTests.cs index 4d16db12..aa9bd8f6 100644 --- a/src/XTerm.NET.Tests/Graphics/SixelDecoderTests.cs +++ b/src/XTerm.NET.Tests/Graphics/SixelDecoderTests.cs @@ -23,13 +23,26 @@ public class SixelDecoderTests /// Background select 0: pixels left unset take the terminal background. private const int OpaqueBackground = 0; + private static readonly System.Runtime.CompilerServices.ConditionalWeakTable> Seen = new(); + private static Terminal Fresh(Action? configure = null) { var options = new TerminalOptions { Cols = 40, Rows = 10 }; configure?.Invoke(options); - return new Terminal(options); + var terminal = new Terminal(options); + + // The buffer keeps a Sixel's pixels, not the picture, so the decoded picture is caught as + // it is placed. + var seen = new List(); + terminal.SixelPlaced += seen.Add; + Seen.Add(terminal, seen); + return terminal; } + /// The last picture a Sixel placed, or null if none reached the buffer. + private static TerminalImage? Placed(Terminal terminal) + => Seen.TryGetValue(terminal, out var seen) && seen.Count > 0 ? seen[^1] : null; + private static void WriteSixel(Terminal terminal, string body, int backgroundSelect = Transparent) => terminal.Write($"{Esc}P0;{backgroundSelect};0q{body}{St}"); @@ -38,7 +51,7 @@ private static void WriteSixel(Terminal terminal, string body, int backgroundSel { var terminal = Fresh(configure); WriteSixel(terminal, body, backgroundSelect); - return (terminal.Buffer.Lines[0]!.TryGetImageAt(0, out var __i1) ? __i1 : null); + return Placed(terminal); } private static TerminalImage Decode(string body, int backgroundSelect = Transparent, @@ -232,7 +245,7 @@ public void An_abandoned_payload_produces_no_image() // CAN mid-payload: the sequence is dropped rather than terminated. terminal.Write($"{Esc}P0;1;0q#0;2;100;0;0!20~\u0018"); - Assert.Null((terminal.Buffer.Lines[0]!.TryGetImageAt(0, out var __i2) ? __i2 : null)); + Assert.Null(Placed(terminal)); } [Fact] @@ -241,7 +254,7 @@ public void An_empty_payload_produces_no_image() var terminal = Fresh(); terminal.Write($"{Esc}P0;1;0q{St}"); - Assert.Null((terminal.Buffer.Lines[0]!.TryGetImageAt(0, out var __i3) ? __i3 : null)); + Assert.Null(Placed(terminal)); } /// @@ -283,8 +296,8 @@ public void A_payload_split_across_writes_decodes_the_same() split.Write("0;0;0!4~-#1;2;0"); split.Write($";0;100!4~{St}"); - var a = whole.Buffer.Lines[0]!.TryGetImageAt(0, out var wholeImage) ? wholeImage : null; - var b = split.Buffer.Lines[0]!.TryGetImageAt(0, out var splitImage) ? splitImage : null; + var a = Placed(whole); + var b = Placed(split); Assert.NotNull(a); Assert.NotNull(b); @@ -300,7 +313,7 @@ public void Sixel_can_be_switched_off_entirely() var terminal = Fresh(o => o.SixelEnabled = false); WriteSixel(terminal, "#0;2;100;0;0~"); - Assert.Null((terminal.Buffer.Lines[0]!.TryGetImageAt(0, out var __i4) ? __i4 : null)); + Assert.Null(Placed(terminal)); } [Fact] diff --git a/src/XTerm.NET.Tests/Graphics/SixelPlacementTests.cs b/src/XTerm.NET.Tests/Graphics/SixelPlacementTests.cs index 18cd6364..657c8ea1 100644 --- a/src/XTerm.NET.Tests/Graphics/SixelPlacementTests.cs +++ b/src/XTerm.NET.Tests/Graphics/SixelPlacementTests.cs @@ -19,6 +19,22 @@ public class SixelPlacementTests /// Four pixels wide, twelve tall: two cells across, four down. private const string TwoByFourCells = "#0;2;100;0;0!4~-!4~"; + /// + /// The same size, with each cell row its own colour -- red, green, blue, yellow from the top -- + /// so the colour a screen row shows says which band of the picture landed there. 'F' sets a + /// sixel's top three pixels and 'w' its bottom three; a cell is three pixels tall. + /// + private const string Striped = "#0;2;100;0;0!4F$#1;2;0;100;0!4w-#2;2;0;0;100!4F$#3;2;100;100;0!4w"; + + private static readonly (byte R, byte G, byte B, byte A)[] Stripes = + { + (255, 0, 0, 255), (0, 255, 0, 255), (0, 0, 255, 255), (255, 255, 0, 255), + }; + + /// Which stripe of a screen position shows, or -1 for none. + private static int StripeOn(Terminal terminal, int col, int screenRow) + => ImageAssertions.SixelPixel(terminal, col, screenRow) is { } pixel ? Array.IndexOf(Stripes, pixel) : -1; + private static Terminal Fresh(int rows = 10, Action? configure = null) { var options = new TerminalOptions @@ -39,30 +55,21 @@ private static BufferCell Cell(Terminal terminal, int col, int screenRow) => terminal.Buffer.Lines[terminal.Buffer.YBase + screenRow]![col]; [Fact] - public void An_image_covers_one_cell_per_tile() + public void An_image_paints_each_cell_it_covers_with_its_own_band() { var terminal = Fresh(); - WriteSixel(terminal); - - var image = ImageAssertions.ImageAt(terminal, 0, 0); - Assert.NotNull(image); - Assert.Equal(2, image!.Cols); - Assert.Equal(4, image.Rows); + WriteSixel(terminal, Striped); - // One run per line, each covering the picture's full width and carrying its own slice of - // the source — which is what the per-cell tile grid said, expressed once per row instead of - // once per cell. + // Two columns by four rows, each row showing the band of the picture that belongs there, + // and every line's strip at the cell size the picture was laid out against. for (int row = 0; row < 4; row++) { - var placement = ImageAssertions.PlacementAt(terminal, 0, row); - Assert.NotNull(placement); - Assert.True(ReferenceEquals(ImageAssertions.ImageAt(terminal, 0, row), image), - $"row {row} should show part of the image"); - - Assert.Equal(0, placement!.Value.Column); - Assert.Equal(2, placement.Value.Cols); - Assert.Equal(0, placement.Value.SrcX); - Assert.Equal(row * terminal.Options.CellHeightPixels, placement.Value.SrcY); + for (int col = 0; col < 2; col++) + Assert.Equal(row, StripeOn(terminal, col, row)); + + var strip = ImageAssertions.StripOn(terminal, row)!; + Assert.Equal(terminal.Options.CellWidthPixels, strip.CellWidth); + Assert.Equal(terminal.Options.CellHeightPixels, strip.CellHeight); } } @@ -72,8 +79,9 @@ public void Cells_beyond_the_image_are_left_alone() var terminal = Fresh(); WriteSixel(terminal); - Assert.Null(ImageAssertions.ImageAt(terminal, 2, 0)); - Assert.Null(ImageAssertions.ImageAt(terminal, 0, 4)); + Assert.False(ImageAssertions.IsSixelAt(terminal, 2, 0)); + Assert.False(ImageAssertions.IsSixelAt(terminal, 0, 4)); + Assert.Null(ImageAssertions.StripOn(terminal, 4)); } [Fact] @@ -83,9 +91,9 @@ public void An_image_starts_at_the_cursor() terminal.Write($"{Esc}[3;6H"); // row 3, column 6, one-based WriteSixel(terminal); - Assert.True(ReferenceEquals(ImageAssertions.ImageAt(terminal, 5, 2), ImageAssertions.ImageAt(terminal, 6, 2))); - Assert.NotNull(ImageAssertions.ImageAt(terminal, 5, 2)); - Assert.Null(ImageAssertions.ImageAt(terminal, 4, 2)); + Assert.True(ImageAssertions.IsSixelAt(terminal, 5, 2)); + Assert.True(ImageAssertions.IsSixelAt(terminal, 6, 2)); + Assert.False(ImageAssertions.IsSixelAt(terminal, 4, 2)); } [Fact] @@ -133,8 +141,8 @@ public void Decsdm_pins_the_image_to_the_top_left_and_leaves_the_cursor_alone() terminal.Write($"{Esc}[3;6H"); WriteSixel(terminal); - Assert.NotNull(ImageAssertions.ImageAt(terminal, 0, 0)); - Assert.Null(ImageAssertions.ImageAt(terminal, 5, 2)); + Assert.True(ImageAssertions.IsSixelAt(terminal, 0, 0)); + Assert.False(ImageAssertions.IsSixelAt(terminal, 5, 2)); Assert.Equal(5, terminal.Buffer.X); Assert.Equal(2, terminal.Buffer.Y); } @@ -148,7 +156,7 @@ public void Decsdm_reset_restores_the_scrolling_behaviour() terminal.Write($"{Esc}[3;6H"); WriteSixel(terminal); - Assert.NotNull(ImageAssertions.ImageAt(terminal, 5, 2)); + Assert.True(ImageAssertions.IsSixelAt(terminal, 5, 2)); } /// @@ -159,15 +167,11 @@ public void An_image_that_runs_past_the_bottom_scrolls_the_screen() { var terminal = Fresh(rows: 5); terminal.Write($"{Esc}[4;1H"); // last-but-one row - WriteSixel(terminal); + WriteSixel(terminal, Striped); // Four image rows plus the cursor's own row need five: the screen scrolled until they fit. for (int row = 0; row < 4; row++) - { - var placement = ImageAssertions.PlacementAt(terminal, 0, row); - Assert.NotNull(placement); - Assert.Equal(row * terminal.Options.CellHeightPixels, placement!.Value.SrcY); - } + Assert.Equal(row, StripeOn(terminal, 0, row)); Assert.Equal(0, terminal.Buffer.X); Assert.Equal(4, terminal.Buffer.Y); @@ -177,12 +181,12 @@ public void An_image_that_runs_past_the_bottom_scrolls_the_screen() public void An_image_taller_than_the_screen_keeps_its_last_rows() { var terminal = Fresh(rows: 3); - WriteSixel(terminal); + WriteSixel(terminal, Striped); // Three rows of screen, four of image, and the cursor still needs a row of its own below // it -- so the picture scrolled up until its last two rows and the cursor fit. - Assert.Equal(2 * terminal.Options.CellHeightPixels, ImageAssertions.PlacementAt(terminal, 0, 0)!.Value.SrcY); - Assert.Equal(3 * terminal.Options.CellHeightPixels, ImageAssertions.PlacementAt(terminal, 0, 1)!.Value.SrcY); + Assert.Equal(2, StripeOn(terminal, 0, 0)); + Assert.Equal(3, StripeOn(terminal, 0, 1)); Assert.Equal(2, terminal.Buffer.Y); } @@ -191,11 +195,11 @@ public void Decsdm_clips_a_tall_image_instead_of_scrolling() { var terminal = Fresh(rows: 3); terminal.Write($"{Esc}[?80h"); - WriteSixel(terminal); + WriteSixel(terminal, Striped); // Pinned at the top, so the first rows are the ones that survive. - Assert.Equal(0 * terminal.Options.CellHeightPixels, ImageAssertions.PlacementAt(terminal, 0, 0)!.Value.SrcY); - Assert.Equal(2 * terminal.Options.CellHeightPixels, ImageAssertions.PlacementAt(terminal, 0, 2)!.Value.SrcY); + Assert.Equal(0, StripeOn(terminal, 0, 0)); + Assert.Equal(2, StripeOn(terminal, 0, 2)); } [Fact] @@ -203,10 +207,13 @@ public void An_image_is_clipped_at_the_right_edge() { var terminal = Fresh(configure: o => o.Cols = 6); terminal.Write($"{Esc}[1;6H"); // one column from the right edge - WriteSixel(terminal); + WriteSixel(terminal, Striped); - Assert.NotNull(ImageAssertions.ImageAt(terminal, 5, 0)); - Assert.Equal(0 * terminal.Options.CellWidthPixels, ImageAssertions.PlacementAt(terminal, 5, 0)!.Value.SrcX); + // The picture's first column lands in the last cell; the rest runs past the edge, kept at + // its natural width for a wider window but not shown on this one. + Assert.True(ImageAssertions.IsSixelAt(terminal, 5, 0)); + Assert.Equal(0, StripeOn(terminal, 5, 0)); + Assert.False(ImageAssertions.IsSixelAt(terminal, 6, 0)); } /// @@ -243,18 +250,18 @@ public void Placing_an_image_drops_the_lines_render_cache() } [Fact] - public void Two_images_do_not_share_tiles() + public void A_second_image_paints_below_the_first_and_leaves_it_alone() { var terminal = Fresh(); - WriteSixel(terminal); - WriteSixel(terminal); - - var first = ImageAssertions.ImageAt(terminal, 0, 0); - var second = ImageAssertions.ImageAt(terminal, 0, 4); + WriteSixel(terminal, Striped); + WriteSixel(terminal, Striped); - Assert.NotNull(first); - Assert.NotNull(second); - Assert.False(ReferenceEquals(first, second)); - Assert.NotEqual(first!.Id, second!.Id); + // The first picture's rows are untouched by the second, which starts where the cursor + // was left: the row below the first. + for (int row = 0; row < 4; row++) + { + Assert.Equal(row, StripeOn(terminal, 0, row)); + Assert.Equal(row, StripeOn(terminal, 0, row + 4)); + } } } diff --git a/src/XTerm.NET.Tests/ITerm2OscTests.cs b/src/XTerm.NET.Tests/ITerm2OscTests.cs index a92b5645..533fc9fb 100644 --- a/src/XTerm.NET.Tests/ITerm2OscTests.cs +++ b/src/XTerm.NET.Tests/ITerm2OscTests.cs @@ -21,12 +21,16 @@ public class ITerm2OscTests public void File_InlinePng_IsDecodedAndPlaced() { var terminal = CreateTerminal(); + XTerm.Graphics.TerminalImage? image = null; + terminal.SixelPlaced += placed => image = placed; + terminal.Write(Osc($"File=inline=1:{Png}")); - var image = ImageAssertions.ImageAt(terminal, 0, 0); + // An inline image is content, as a Sixel is: decoded, then painted into the line. Assert.NotNull(image); Assert.Equal(5, image!.PixelWidth); Assert.Equal(3, image.PixelHeight); + Assert.True(ImageAssertions.IsSixelAt(terminal, 0, 0)); } [Fact] diff --git a/src/XTerm.NET.Tests/XtGetTcapTests.cs b/src/XTerm.NET.Tests/XtGetTcapTests.cs index 22ab0645..66cb82d3 100644 --- a/src/XTerm.NET.Tests/XtGetTcapTests.cs +++ b/src/XTerm.NET.Tests/XtGetTcapTests.cs @@ -269,7 +269,7 @@ public void A_sixel_payload_is_not_mistaken_for_a_capability_request() terminal.Write($"{Esc}P0;1;0q{SixelImage}{St}"); Assert.Empty(replies); - Assert.NotNull(ImageAssertions.ImageAt(terminal, 0, 0)); + Assert.True(ImageAssertions.IsSixelAt(terminal, 0, 0)); } [Fact] @@ -292,6 +292,6 @@ public void An_image_after_a_capability_request_is_still_decoded() terminal.Write($"{Esc}P0;1;0q{SixelImage}{St}"); - Assert.NotNull(ImageAssertions.ImageAt(terminal, 0, 0)); + Assert.True(ImageAssertions.IsSixelAt(terminal, 0, 0)); } } diff --git a/src/XTerm.NET/Buffer/BufferLine.cs b/src/XTerm.NET/Buffer/BufferLine.cs index 26797b4a..bc4a0511 100644 --- a/src/XTerm.NET/Buffer/BufferLine.cs +++ b/src/XTerm.NET/Buffer/BufferLine.cs @@ -47,6 +47,22 @@ public class BufferLine : IEnumerable /// with no eviction pass and nothing to keep in step with a buffer that scrolls. /// private List? _images; + + /// + /// The Sixel pixels this line shows, or null -- which is every line no Sixel has landed on. + /// + /// + /// Sixel is kept as the pixels it painted rather than as pictures: see + /// . Kitty pictures are overlays with identities and z-order and + /// stay as runs in . + /// + private Graphics.SixelStrip? _sixel; + + /// + /// Whether is shared with a copy of this line, so the next write must take a + /// private copy first. Set on both sides by and . + /// + private bool _sixelShared; private int _length; private bool _isWrapped; private LineAttribute _lineAttribute; @@ -179,6 +195,10 @@ public void SetCell(int index, ref BufferCell cell) if (_placements is not null) SplitPlacementsAt(index); + // And the Sixel pixels in that cell, the same rule. + if (_sixel is not null) + ClearSixel(index, 1); + Cache = null; } } @@ -214,6 +234,8 @@ public void SetSingleWidthRun(int index, ReadOnlySpan text, AttributeData // One field test on a line with no pictures, which is nearly every line. if (_placements is not null) SplitPlacementsOver(index, text.Length); + if (_sixel is not null) + ClearSixel(index, text.Length); Cache = null; } @@ -240,6 +262,8 @@ public void SetSingleWidthRun(int index, ReadOnlySpan ascii, AttributeData // treatment SetCell gives them. if (_placements is not null) SplitPlacementsOver(index, ascii.Length); + if (_sixel is not null) + ClearSixel(index, ascii.Length); Cache = null; } @@ -324,6 +348,8 @@ public void Fill(BufferCell fillCell, int startCol = 0, int endCol = -1) if (_placements is not null) SplitPlacementsOver(startCol, Math.Max(0, Math.Min(endCol, _length) - startCol), includeOverlays: true); + if (_sixel is not null) + ClearSixel(startCol, Math.Max(0, Math.Min(endCol, _length) - startCol)); // And a LINK, unlike a mark. A mark records a position in the history and survives the // shell redrawing its prompt; a link is a property of its text, and text that has been @@ -381,9 +407,33 @@ public void CopyCellsFrom(BufferLine src, int srcCol, int destCol, int length, b } } } + + // The Sixel pixels follow their cells: an insert, a delete or a margin scroll that moved the + // text moved the picture it sits in. Kitty runs are anchored overlays and stay put. + if (src._sixel is not null || _sixel is not null) + CopySixelCells(src, srcCol, destCol, length); + Cache = null; } + private void CopySixelCells(BufferLine src, int srcCol, int destCol, int length) + { + var count = Math.Min(length, Math.Min(_length - destCol, src._length - srcCol)); + if (count <= 0 || destCol < 0 || srcCol < 0) + return; + + // Blanks onto a line with no pixels there: nothing to do, and nothing to allocate. + if (src._sixel is null && !SixelPaintedIn(destCol, count)) + return; + + var strip = WritableSixel(src._sixel); + var source = ReferenceEquals(src, this) ? strip : src._sixel; + strip.CopyCells(source, srcCol, destCol, count); + + if (strip.IsEmpty) + DropSixel(); + } + /// /// Translates the line to a string. /// @@ -454,21 +504,101 @@ public void ReplaceCells(int startCol, int endCol, BufferCell fillCell) /// True if the line held any, which is also the signal that it needs repainting. public bool ClearImages() { - if (_placements is null && _images is null) + if (_placements is null && _images is null && _sixel is null) return false; // Nothing to clean up in the cells — they never held anything. Releasing the strong // references is what actually frees the pixels. _placements = null; _images = null; + _sixel = null; + _sixelShared = false; Cache = null; return true; } /// - /// Whether this line shows any part of a picture. One field test — a renderer can ask per row. + /// Whether this line shows any part of a picture -- a Kitty run or Sixel pixels. One field test + /// each, so a renderer can ask per row. + /// + public bool HasImages => _placements is { Count: > 0 } || _sixel is not null; + + /// The Sixel pixels this line shows, or null. See . + public Graphics.SixelStrip? Sixel => _sixel; + + /// + /// Paints one cell-row of a Sixel (or an inline image) into this line, from + /// on. The band of the picture painted starts at pixel row + /// . + /// + internal void PaintSixel(Graphics.TerminalImage image, int sourceY, int column) + { + var strip = WritableSixel(null, image.CellWidth, image.CellHeight); + strip.Paint(image, sourceY, column); + Cache = null; + } + + /// Drops this line's Sixel pixels altogether. What the image budget reclaims through. + /// The bytes released, or zero when there were none. + internal long DropSixel() + { + if (_sixel is null) + return 0; + + var bytes = _sixel.ByteCount; + _sixel = null; + _sixelShared = false; + Cache = null; + return bytes; + } + + /// Clears the Sixel pixels in a span of cells: something printed or erased there replaced them. + private void ClearSixel(int column, int count) + { + if (_sixel is null || count <= 0 || !SixelPaintedIn(column, count)) + return; + + var strip = WritableSixel(null); + strip.Clear(column, count); + if (strip.IsEmpty) + DropSixel(); + } + + private bool SixelPaintedIn(int column, int count) + { + if (_sixel is null) + return false; + + var end = Math.Min(_sixel.Columns, column + count); + for (var c = Math.Max(0, column); c < end; c++) + { + if (_sixel.HasPixelsAt(c)) + return true; + } + + return false; + } + + /// + /// This line's strip, made private if it was shared and created if there was none -- at the cell + /// size of , or the one given. /// - public bool HasImages => _placements is { Count: > 0 }; + private Graphics.SixelStrip WritableSixel(Graphics.SixelStrip? like, int cellWidth = 0, int cellHeight = 0) + { + if (_sixel is null) + { + _sixel = Graphics.SixelStrip.Create( + like?.CellWidth ?? cellWidth, like?.CellHeight ?? cellHeight, _length); + _sixelShared = false; + } + else if (_sixelShared) + { + _sixel = _sixel.Copy(); + _sixelShared = false; + } + + return _sixel; + } /// Whether this line carries any shell-integration mark. public bool HasMarks => _marks is { Count: > 0 }; @@ -1139,6 +1269,14 @@ public BufferLine Clone() newLine._placements = new List(_placements); newLine._images = _images is null ? null : new List(_images); } + + // Shared, not copied: whichever of the two writes first takes its own copy. + if (_sixel is not null) + { + newLine._sixel = _sixel; + newLine._sixelShared = true; + _sixelShared = true; + } for (int i = 0; i < _length; i++) { newLine._cells[i] = _cells[i]; @@ -1178,6 +1316,12 @@ public void CopyFrom(BufferLine line) ? null : new List(line._images); + // Shared with the source until one of them writes, as in Clone. + _sixel = line._sixel; + _sixelShared = _sixel is not null; + if (_sixel is not null) + line._sixelShared = true; + // Assigned, not OR-ed: this REPLACES the line's contents rather than adding to them, so // the latch belongs to the incoming cells. A recycled scrollback line that once held a // wide character would otherwise carry that latch forever. diff --git a/src/XTerm.NET/Graphics/SixelStrip.cs b/src/XTerm.NET/Graphics/SixelStrip.cs new file mode 100644 index 00000000..41947ddc --- /dev/null +++ b/src/XTerm.NET/Graphics/SixelStrip.cs @@ -0,0 +1,327 @@ +namespace XTerm.Graphics; + +/// +/// The Sixel pixels one line shows: a strip one cell tall, painted by every Sixel that lands on the +/// line and erased cell by cell by whatever is printed or erased over it. +/// +/// +/// Pixels, not pictures. A Sixel is content: it paints the cells it lands on, a later +/// Sixel paints over it, and printing over a cell replaces that part of it. So the line keeps the +/// RESULT -- one strip of pixels -- rather than every picture that ever arrived and a record of which +/// slivers of each are still showing. A program that repaints small regions over and over (a cursor, +/// a hover, a diffing framebuffer) costs one strip per line however many Sixels it sends, where +/// keeping the pictures grew thousands of fragments per screen, every one a separate draw. +/// Layout. BGRA, straight alpha, top row first, exactly like +/// : cells of pixels +/// across and pixels down. A pixel nothing has painted is fully transparent, +/// so the cell beneath shows through it. is the widest anything has painted and +/// is never clipped to the line, so a terminal widened later shows more rather than having lost it -- +/// the renderer draws min(Columns, line width), as it does for a picture. +/// One scale per strip. A Sixel is laid out against the cell size in force when it +/// arrives. A strip painted at another size is resampled to the new one first, so every pixel in it +/// agrees on how big a cell is and a renderer maps it to the grid with a single scale. +/// Sharing. A copy of a line (, +/// ) shares its strip, and whichever line writes first takes a +/// private copy, so a frame captured for a renderer costs a reference rather than the pixels. Nothing +/// outside this assembly can write to a strip. +/// Change. is taken from one counter for every strip and moves +/// after each change is complete, so a renderer holding an upload keyed on it knows the upload is +/// current, and one that read the pixels while they were being written sees the version move and +/// uploads again. +/// +public sealed class SixelStrip +{ + private static long _versionCounter; + + private byte[] _pixels; + private bool[] _painted; + private int _paintedCount; + + private SixelStrip(int cellWidth, int cellHeight, int columns) + { + CellWidth = cellWidth; + CellHeight = cellHeight; + Columns = columns; + _pixels = new byte[columns * cellWidth * cellHeight * TerminalImage.BytesPerPixel]; + _painted = new bool[columns]; + Version = NextVersion(); + } + + /// Width of one cell in this strip's pixels. + public int CellWidth { get; private set; } + + /// Height of one cell, which is the strip's height, in pixels. + public int CellHeight { get; private set; } + + /// How many cells across the strip holds pixels for. + public int Columns { get; private set; } + + /// The strip's width in pixels. + public int PixelWidth => Columns * CellWidth; + + /// The strip's height in pixels: one cell. + public int PixelHeight => CellHeight; + + /// Bytes per row of . + public int Stride => PixelWidth * TerminalImage.BytesPerPixel; + + /// The pixels, BGRA with straight alpha, top row first. See the remarks on the class. + public ReadOnlyMemory Pixels => _pixels; + + /// Bytes the pixels occupy. + public long ByteCount => _pixels.LongLength; + + /// Changes after every change to the pixels, and is never reused by another strip. + public long Version { get; private set; } + + /// Whether anything has painted since it was last cleared. + public bool HasPixelsAt(int column) => column >= 0 && column < Columns && _painted[column]; + + /// + /// One past the rightmost painted cell: how much of the strip a renderer needs to draw. Zero when + /// nothing is painted. + /// + public int PaintedColumns + { + get + { + for (var c = Columns - 1; c >= 0; c--) + { + if (_painted[c]) + return c + 1; + } + + return 0; + } + } + + /// Whether every column is clear, in which case the line drops the strip. + internal bool IsEmpty => _paintedCount == 0; + + private static long NextVersion() => Interlocked.Increment(ref _versionCounter); + + /// A strip ready to be painted at the given cell size. + internal static SixelStrip Create(int cellWidth, int cellHeight, int columns) + => new(Math.Max(1, cellWidth), Math.Max(1, cellHeight), Math.Max(1, columns)); + + /// A private copy, for a line that shared this strip and is about to write to it. + internal SixelStrip Copy() + { + var copy = new SixelStrip(CellWidth, CellHeight, Columns); + System.Buffer.BlockCopy(_pixels, 0, copy._pixels, 0, _pixels.Length); + Array.Copy(_painted, copy._painted, _painted.Length); + copy._paintedCount = _paintedCount; + return copy; + } + + /// + /// Paints one cell-row of -- the band starting at pixel row + /// -- into this strip from on. + /// + /// + /// A fully transparent source pixel leaves the strip alone, which is Sixel's own rule: a + /// picture drawn with background select 1 lets whatever was there show through its unset pixels. + /// A fully opaque one replaces the strip's pixel. Anything between -- an inline PNG's soft edge -- + /// is blended over it. + /// The cells the picture covers are marked painted, the last one included even where the + /// picture ends partway through it: that cell shows part of the picture. + /// + internal void Paint(TerminalImage image, int sourceY, int column) + { + if (column < 0 || sourceY < 0 || sourceY >= image.PixelHeight) + return; + + if (image.CellWidth != CellWidth || image.CellHeight != CellHeight) + Resample(image.CellWidth, image.CellHeight); + + var imageCols = (image.PixelWidth + CellWidth - 1) / CellWidth; + EnsureColumns(column + imageCols); + + var rows = Math.Min(CellHeight, image.PixelHeight - sourceY); + var width = image.PixelWidth; + var source = image.CurrentPixels.Span; + var sourceStride = image.Stride; + var stride = Stride; + var destX = column * CellWidth * TerminalImage.BytesPerPixel; + + for (var y = 0; y < rows; y++) + { + var src = source.Slice((sourceY + y) * sourceStride, width * TerminalImage.BytesPerPixel); + var dst = _pixels.AsSpan(y * stride + destX, width * TerminalImage.BytesPerPixel); + + for (var x = 0; x < src.Length; x += TerminalImage.BytesPerPixel) + { + var alpha = src[x + 3]; + if (alpha == 0) + continue; + + if (alpha == 255 || dst[x + 3] == 0) + { + src.Slice(x, TerminalImage.BytesPerPixel).CopyTo(dst.Slice(x)); + continue; + } + + Blend(src.Slice(x, TerminalImage.BytesPerPixel), dst.Slice(x, TerminalImage.BytesPerPixel)); + } + } + + for (var c = column; c < column + imageCols; c++) + MarkPainted(c); + + Version = NextVersion(); + } + + /// Clears cells from : printing or erasing there replaced the picture. + /// Whether anything was cleared. + internal bool Clear(int column, int count) + { + var start = Math.Max(0, column); + var end = Math.Min(Columns, column + count); + var cleared = false; + + for (var c = start; c < end; c++) + { + if (!_painted[c]) + continue; + + ClearCell(c); + _painted[c] = false; + _paintedCount--; + cleared = true; + } + + if (cleared) + Version = NextVersion(); + return cleared; + } + + /// + /// Copies cells of (or blanks, when it is null) + /// to -- the pixels following their cells through an insert, a + /// delete or a margin scroll. Overlapping ranges of one strip copy as if through a buffer. + /// + internal void CopyCells(SixelStrip? source, int sourceColumn, int destColumn, int count) + { + if (count <= 0) + return; + + if (source is not null && (source.CellWidth != CellWidth || source.CellHeight != CellHeight)) + { + source = source.Copy(); + source.Resample(CellWidth, CellHeight); + } + + EnsureColumns(destColumn + count); + + // Through a scratch copy when the ranges are one strip's: simpler than choosing a direction + // per row, and a shift within a line is a keystroke's worth of work. + var from = ReferenceEquals(source, this) ? Copy() : source; + var cellBytes = CellWidth * TerminalImage.BytesPerPixel; + + for (var i = 0; i < count; i++) + { + var dest = destColumn + i; + var src = sourceColumn + i; + var has = from is not null && from.HasPixelsAt(src); + + if (!has) + { + if (_painted[dest]) + { + ClearCell(dest); + _painted[dest] = false; + _paintedCount--; + } + continue; + } + + for (var y = 0; y < CellHeight; y++) + System.Buffer.BlockCopy(from!._pixels, y * from.Stride + src * cellBytes, _pixels, y * Stride + dest * cellBytes, cellBytes); + + MarkPainted(dest); + } + + Version = NextVersion(); + } + + private void MarkPainted(int column) + { + if (_painted[column]) + return; + _painted[column] = true; + _paintedCount++; + } + + private void ClearCell(int column) + { + var cellBytes = CellWidth * TerminalImage.BytesPerPixel; + for (var y = 0; y < CellHeight; y++) + Array.Clear(_pixels, y * Stride + column * cellBytes, cellBytes); + } + + /// Widens the strip to hold at least cells; new cells are clear. + private void EnsureColumns(int columns) + { + if (columns <= Columns) + return; + + var oldStride = Stride; + var grown = new byte[columns * CellWidth * CellHeight * TerminalImage.BytesPerPixel]; + var newStride = columns * CellWidth * TerminalImage.BytesPerPixel; + for (var y = 0; y < CellHeight; y++) + System.Buffer.BlockCopy(_pixels, y * oldStride, grown, y * newStride, oldStride); + + Array.Resize(ref _painted, columns); + _pixels = grown; + Columns = columns; + } + + /// + /// Rescales every pixel to a new cell size, nearest neighbour: what was painted when cells were + /// another size keeps its place in the grid. Only reached when the cell size has changed. + /// + private void Resample(int cellWidth, int cellHeight) + { + cellWidth = Math.Max(1, cellWidth); + cellHeight = Math.Max(1, cellHeight); + + var oldWidth = PixelWidth; + var oldHeight = CellHeight; + var oldStride = Stride; + var width = Columns * cellWidth; + var stride = width * TerminalImage.BytesPerPixel; + var resampled = new byte[stride * cellHeight]; + + for (var y = 0; y < cellHeight; y++) + { + var sy = Math.Min(oldHeight - 1, y * oldHeight / cellHeight); + for (var x = 0; x < width; x++) + { + var sx = Math.Min(oldWidth - 1, x * oldWidth / width); + System.Buffer.BlockCopy(_pixels, sy * oldStride + sx * TerminalImage.BytesPerPixel, + resampled, y * stride + x * TerminalImage.BytesPerPixel, TerminalImage.BytesPerPixel); + } + } + + _pixels = resampled; + CellWidth = cellWidth; + CellHeight = cellHeight; + } + + /// Source over destination, both straight alpha. + private static void Blend(ReadOnlySpan src, Span dst) + { + var sa = src[3] / 255f; + var da = dst[3] / 255f; + var outA = sa + da * (1 - sa); + if (outA <= 0) + { + dst.Clear(); + return; + } + + for (var i = 0; i < 3; i++) + dst[i] = (byte)Math.Clamp((int)Math.Round((src[i] * sa + dst[i] * da * (1 - sa)) / outA), 0, 255); + dst[3] = (byte)Math.Clamp((int)Math.Round(outA * 255), 0, 255); + } +} diff --git a/src/XTerm.NET/InputHandler.KittyGraphics.cs b/src/XTerm.NET/InputHandler.KittyGraphics.cs index 35a6f1e8..fccb1144 100644 --- a/src/XTerm.NET/InputHandler.KittyGraphics.cs +++ b/src/XTerm.NET/InputHandler.KittyGraphics.cs @@ -989,19 +989,19 @@ private void ReplyToKitty(Graphics.KittyCommand command, Graphics.KittyError err } /// - /// Writes an image into the buffer as one run per line. + /// Writes an image into the buffer, a cell-row at a time: Sixel as pixels painted into each + /// line's strip, Kitty as one run per line. /// /// - /// A picture spanning several rows becomes several s, - /// one per line, each carrying its own slice of the source. The line owns the run and the image, - /// so scrolling, scrollback eviction and ownership all keep working without anything being - /// written into cells — and a resize does nothing to a picture at all. - /// Cells are not touched. That is the whole difference between the two protocols here: - /// Sixel is CONTENT, so printing over it replaces that part of the picture and - /// does it explicitly; Kitty is an OVERLAY the - /// z-index orders against the text, so a picture placed over a character hides it while it is - /// there and reveals it again when it is deleted. Blanking the cell would make that - /// irreversible. + /// The two protocols differ in what a picture IS. Sixel is CONTENT: it paints the cells it + /// lands on, a later Sixel paints over it, and printing over a cell replaces that part -- so each + /// line keeps the pixels themselves, in a , and the picture that + /// delivered them is not kept at all. Kitty is an OVERLAY with an identity and a z-index: a + /// picture placed over a character hides it while it is there and reveals it again when it is + /// deleted, so it is kept as several s, one per line, each + /// carrying its own slice of the source. + /// Either way the line owns what it shows, so scrolling, scrollback eviction and ownership + /// keep working without anything written into cells, and a resize does nothing to a picture. /// private void PlaceImage(Graphics.ImagePlacement placement, Graphics.PlacementKind kind, @@ -1019,6 +1019,9 @@ private void PlaceImage(Graphics.ImagePlacement placement, var lastRowDrawn = row; + if (kind == Graphics.PlacementKind.Sixel) + _terminal.NoteSixelPlaced(placement.Image); + // One serial for the placement, shared by every row of it, so a delete aimed at any one // cell can find and remove the whole picture. var serial = Graphics.LinePlacement.NextSerial(); @@ -1040,14 +1043,21 @@ private void PlaceImage(Graphics.ImagePlacement placement, if (line is null) break; - // Sixel is content, so a picture landing on cells that already show one replaces that - // part of it, exactly as printing a character there does. Without this every Sixel a - // program draws over an earlier one is kept beneath it: a line in the alternate screen - // never dies, so an application repainting small regions (a cursor, a hover highlight) - // grew an unbounded stack of runs per line, each drawn every frame and each pinning its - // picture in memory. Kitty placements are overlays ordered by z-index and are left alone. + // Sixel is content: its pixels are painted into the line, over whatever Sixel was there, + // and nothing about the picture is kept. Keeping pictures and splitting the ones + // underneath grew thousands of fragments per screen under a program that repaints small + // regions, every one of them its own draw. if (kind == Graphics.PlacementKind.Sixel) - line.SplitPlacementsOver(startCol, placement.Cols); + { + if (placement.TryGetTileLayout(0, tileRow, out _, out var sixelY, out _, out _, + out _, out _, out _, out _)) + line.PaintSixel(placement.Image, sixelY, startCol); + + lastRowDrawn = row; + if (tileRow < placement.Rows - 1) + row++; + continue; + } // One run per line. Cols is the picture's NATURAL width — deliberately NOT clipped to // the terminal, so a window widened later reveals more of the picture rather than diff --git a/src/XTerm.NET/Terminal.cs b/src/XTerm.NET/Terminal.cs index feda20ec..d88aca16 100644 --- a/src/XTerm.NET/Terminal.cs +++ b/src/XTerm.NET/Terminal.cs @@ -1367,6 +1367,18 @@ public void ScrollLines(int lines) /// picture. What that trades away is exactness: the buffer can sit up to one budget over before the /// sweep, which is a ceiling on overshoot rather than an unbounded one. /// + /// + /// Raised with each Sixel or inline image as it is painted into the buffer. + /// + /// + /// The buffer keeps what a Sixel painted, not the picture that painted it (see + /// ), so this is the one place the decoded picture can be seen + /// whole -- which is what the decoder's tests need to see. + /// + internal event Action? SixelPlaced; + + internal void NoteSixelPlaced(Graphics.TerminalImage image) => SixelPlaced?.Invoke(image); + internal void NoteImagePlaced(Graphics.TerminalImage image) { if (Options.MaxImageBytes <= 0) @@ -1584,13 +1596,20 @@ internal void EnforceImageBudget() return; var live = CollectLiveImages(); - long total = 0; + long total = SixelBytes(_normalBuffer) + SixelBytes(_altBuffer); foreach (var image in live) total += image.ByteCount; if (total <= budget) return; + // Sixel pixels in the scrollback go first, oldest line first: they are the history furthest + // from the screen, and a strip has no picture behind it to keep. What is on either screen is + // never taken here -- it is at most one screen of pixels. + total -= DropScrollbackSixels(_normalBuffer, total - budget); + if (total <= budget) + return; + var doomed = new HashSet(); foreach (var image in live.OrderBy(i => i.Id)) { @@ -1607,6 +1626,36 @@ internal void EnforceImageBudget() DropImages(_altBuffer, doomed); } + /// The bytes of Sixel pixels a buffer holds, scrollback and screen. + private static long SixelBytes(Buffer.TerminalBuffer? buffer) + { + if (buffer is null) + return 0; + + long bytes = 0; + for (int i = 0; i < buffer.Lines.Length; i++) + { + if (buffer.Lines[i]?.Sixel is { } strip) + bytes += strip.ByteCount; + } + + return bytes; + } + + /// Drops scrollback Sixel strips, oldest first, until bytes are freed. + /// The bytes freed. + private static long DropScrollbackSixels(Buffer.TerminalBuffer? buffer, long excess) + { + if (buffer is null) + return 0; + + long freed = 0; + for (int i = 0; i < buffer.YBase && i < buffer.Lines.Length && freed < excess; i++) + freed += buffer.Lines[i]?.DropSixel() ?? 0; + + return freed; + } + private HashSet CollectLiveImages() { var live = new HashSet(); diff --git a/src/XTerm.NET/XTerm.NET.csproj b/src/XTerm.NET/XTerm.NET.csproj index 95f19a07..8d9d6ea6 100644 --- a/src/XTerm.NET/XTerm.NET.csproj +++ b/src/XTerm.NET/XTerm.NET.csproj @@ -7,8 +7,8 @@ XTerm - 2.0.8 - 2.0.8.0 + 3.0.0 + 3.0.0.0 Dotnet port of xterm.js, giving you ability to host console applications, with Sixel and Kitty graphics. Iciclecreek Tom Laird-McConnell From 38c60ef639fa56e34acaf77e0b045d4ff67606c2 Mon Sep 17 00:00:00 2001 From: Tom Laird-McConnell Date: Wed, 14 Oct 2026 12:01:05 -0700 Subject: [PATCH 2/2] README: drawing a row with Sixel strips, and reporting cell-size changes The images section now gives the whole draw order for a row (backgrounds, Kitty behind, text, the Sixel strip, Kitty in front) with code, how to key uploads (TerminalImage + FrameSerial, SixelStrip + Version, and the torn read), mapping cells onto the grid after a font or DPI change, and the per-column lookups including Sixel. It also says to call NotifyTextAreaPixelsChanged after changing the cell size: without it an application sizing pictures from the cell size keeps drawing at the old one after a move to another monitor. Co-Authored-By: Claude Opus 5.5 --- README.md | 122 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 90 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index ac2c7590..b982e3e4 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,8 @@ painted them is not kept at all. `line.HasImages` is true for a line with a strip as well as one with Kitty runs. **What a renderer does now.** Per line: Kitty pictures behind the text (negative `ZIndex`), then the -text, then the line's Sixel strip, then Kitty pictures in front. The strip is one blit: +text, then the line's Sixel strip, then Kitty pictures in front — see [Images](#images) for the whole +row. The strip is one blit: ```csharp if (line.Sixel is { } strip) @@ -532,59 +533,116 @@ up to a whole row of the control's height belongs to no row at all. An applicati figure by the row count is told the terminal is taller than it is, sizes a picture to fill it, and the surplus runs off the bottom and scrolls the screen. -**Drawing a line's Sixel pixels.** One blit per line, after the text and before Kitty pictures in -front of it — see [Upgrading to 3.0](#upgrading-to-30) for the code. `strip.PaintedColumns` is how far -anything is painted, so a narrow picture on a wide line does not draw the rest of the row. +**Say when the cell size changes.** A font change, a zoom, or the window moving to a monitor with +another DPI changes how many pixels a cell is while the grid can stay the same — so `Resize` is not +called and nothing tells the application. Update the options and then call +`NotifyTextAreaPixelsChanged()`: -**Drawing the Kitty runs.** This is per *line*, not per cell — a run is one blit, not a row of tiles: +```csharp +terminal.Options.CellWidthPixels = newCellWidth; +terminal.Options.CellHeightPixels = newCellHeight; +terminal.NotifyTextAreaPixelsChanged(); +``` + +That sends an in-band resize report (`CSI 48 ; rows ; cols ; height ; width t`) to any application +that asked for one with mode 2048, and is a no-op otherwise, so it is safe to call whenever the +metrics are recomputed. Without it an application that sizes its pictures from the cell size goes on +drawing at the old one: after a move to a higher-DPI monitor every picture lands smaller than its +cells, with gaps between them. + +**Drawing a row.** Per *line*, not per cell, in this order: + +1. Cell backgrounds — but not under a Kitty picture *behind* the text (negative `ZIndex`), which the + text sits on. +2. Kitty pictures behind the text, back to front. +3. The text. +4. The line's Sixel pixels, if it has any — one blit for the whole line. +5. Kitty pictures in front of the text (`ZIndex >= 0`), back to front. ```csharp var line = buffer.Lines[buffer.YDisp + row]; -// Back to front. A stable sort, so equal depths keep the order they were placed in, which is age. -foreach (var p in line.Placements.OrderBy(p => p.ZIndex)) +// Kitty runs, back to front. A stable sort, so equal depths keep the order they were placed in, +// which is age. +var kitty = line.Placements.OrderBy(p => p.ZIndex).ToList(); + +DrawBackgrounds(line, row); // skipping cells under a negative-z picture +foreach (var p in kitty.Where(p => p.ZIndex < 0)) + DrawKittyRun(line, p, row); +DrawText(line, row); + +if (line.Sixel is { } strip) { - // Pixels are BGRA8888 with straight (unpremultiplied) alpha, top row first. - // Cache your framework's bitmap against the TerminalImage — a ConditionalWeakTable keyed on it - // lets the bitmap die when the image does, with no eviction list to maintain, and two - // placements of one picture share the single upload. - // Matched by id rather than by column: two pictures may overlap the same columns, and a - // column lookup answers with whichever placement comes first, not necessarily this one. + // BGRA8888, straight alpha, top row first: cells of strip.CellWidth x strip.CellHeight pixels. + // PaintedColumns is how far anything is painted, so a narrow picture on a wide line does not + // draw the rest of the row; clip it to the line as well. + var cols = Math.Min(strip.PaintedColumns, line.Length); + DrawImage(StripBitmap(strip), + source: (0, 0, cols * strip.CellWidth, strip.CellHeight), + dest: (0, row * cellHeight, cols * cellWidth, cellHeight)); +} + +foreach (var p in kitty.Where(p => p.ZIndex >= 0)) + DrawKittyRun(line, p, row); + +void DrawKittyRun(BufferLine line, LinePlacement p, int row) +{ + // Matched by id rather than by column: two pictures may overlap the same columns, and a column + // lookup answers with whichever placement comes first, not necessarily this one. var image = line.Images.FirstOrDefault(i => i.Id == p.ImageId); - if (image is null) continue; - var bitmap = _bitmaps.GetOrCreate(image); + if (image is null) return; - DrawImage(bitmap, + DrawImage(PictureBitmap(image), source: (p.SrcX, p.SrcY, p.SrcWidth, p.SrcHeight), dest: ((p.Column * cellWidth) + p.OffsetX, (row * cellHeight) + p.OffsetY, p.Cols * cellWidth, cellHeight)); } ``` -Clip the destination to the line's width and narrow the source by the same proportion — that is what -makes a narrowed window show less of a picture rather than a squashed one. `OffsetX`/`OffsetY` are -the sub-cell shift a Kitty placement asks for with `X=`/`Y=`; they are zero for Sixel and for most -Kitty placements. - -Paint each cell's own background once, underneath the runs, and from the bottom-most run covering it -only — painting it again for a nearer run erases what is behind instead of letting it show through. - -Text goes down *between* the negative z-indices and the rest: see [Draw order](#kitty-graphics) -below for why a negative `ZIndex` means behind the text rather than merely further back. +**Upload once, and key the upload on what changes.** + +- A Kitty picture: cache your bitmap against the `TerminalImage` — a `ConditionalWeakTable` keyed on + it lets the bitmap die when the image does, with no eviction list to maintain, and two placements + of one picture share the single upload. For animation draw `image.CurrentPixels` and upload again + when `image.FrameSerial` changes. +- A Sixel strip: cache against the `SixelStrip` and upload again when `strip.Version` changes. A + strip is painted by the thread writing to the terminal, so read `Version` before copying the + pixels and again after: a copy whose version moved may be torn, so draw it but do not keep it. + A line's copies (`Clone`, `CopyFrom`) share its strip until one of them is written, so a frame + captured for a renderer costs a reference rather than the pixels. + +Draws, not pixels, are what a busy screen costs. A program repainting small regions over and over +leaves at most one strip per row however many Sixels it sends, and a Kitty client that tiles the +screen into blocks costs one draw per block per frame — keep pictures on the GPU between frames +rather than handing the framework a fresh bitmap each time. + +**Mapping onto the grid.** A strip is painted at the cell size in force when its Sixels arrived, and +a picture records the cell size it was laid out against (`image.CellWidth`/`CellHeight`). After a +font or DPI change, stretch them onto the current grid — map cells to cells, as above, rather than +pixels to pixels — so a picture stays locked to the text it was drawn among. + +For Kitty runs, clip the destination to the line's width and narrow the source by the same +proportion — that is what makes a narrowed window show less of a picture rather than a squashed one. +`OffsetX`/`OffsetY` are the sub-cell shift a placement asks for with `X=`/`Y=`; they are zero for most +placements. Paint each cell's own background once, underneath the runs, and from the bottom-most run +covering it only — painting it again for a nearer run erases what is behind instead of letting it +show through. See [Draw order](#kitty-graphics) below for why a negative `ZIndex` means behind the +text rather than merely further back. Cells under a picture hold `" "` as their content, so `TranslateToString` and selection copy yield blanks. -To ask what covers one column rather than walking the list: +To ask what covers one column rather than walking the line: ```csharp -if (line.TryGetPlacementAt(col, out LinePlacement placement)) { … } -if (line.TryGetImageAt(col, out TerminalImage image)) { … } +if (line.Sixel?.HasPixelsAt(col) == true) { … } // Sixel pixels +if (line.TryGetPlacementAt(col, out LinePlacement placement)) { … } // a Kitty run +if (line.TryGetImageAt(col, out TerminalImage image)) { … } // the Kitty picture it shows ``` -Both answer with the *first* placement covering that column, so they are the convenient form for the -ordinary case of one picture at a time — a renderer handling overlapping pictures should walk -`line.Placements` and resolve through `line.Images` as above. +The Kitty lookups answer with the *first* placement covering that column, so they are the convenient +form for the ordinary case of one picture at a time — a renderer handling overlapping pictures should +walk `line.Placements` and resolve through `line.Images` as above. **Options:**