fix: highlight code beyond the first screen after a document opens

CHANGELOG.md

@@ -17,6 +17,7 @@ Leafdown uses lightweight [Keep a Changelog](https://keepachangelog.com/en/1.1.0
 ### Fixed
 
 - Stop larger documents from freezing just after you pause typing. Leafdown converted the whole document back to Markdown 200 ms after each edit, which froze the window for about a third of a second in a 200 KB document of tables and about a second at 500 KB. It now reads the document only when something needs it, such as saving.
+- Open documents with many code blocks without freezing. Leafdown used to highlight every code block before showing the document, so 200 KB of code kept the window unresponsive for about 3.5 seconds and 1 MB for over 20. The document now appears with the code on screen highlighted, in under a second at 200 KB and under 2 seconds at 1 MB, and the rest of its code is highlighted in the background while you read, scroll, and type.
 - Keep typing responsive in documents with many code blocks. Every keystroke used to highlight all of the document's code again, so typing slowed sharply as a document's code grew; now an edit highlights only the code blocks it changes, and typing outside code highlights nothing. Line numbers for code blocks, when shown, are now updated the same way instead of being redrawn for every block on each keystroke.
 - Type faster in documents with many headings. Every keystroke used to work out again which headings could fold and redraw every fold marker and folded section, which took a growing share of each keystroke as headings were added; now an edit updates only the headings and folded sections it can change.
 - Type without long pauses in large documents. Every keystroke used to update the block gutters of the whole document, so in several hundred kilobytes of Markdown with many headings and lists each character could take seconds to appear. The gutters now keep only the blocks in view up to date, and large documents open faster too.

docs/architecture.md

@@ -88,7 +88,7 @@ Shortcut execution follows the layer that owns the interaction. The window-level
 
 Focus ownership follows the same layering. A pointer-opened context popup leaves focus with the editor, which still owns the selection the popup acts on; a keyboard-opened popup takes focus, having no other route in, and returns it to the editor on close rather than leaving it on the document body. ProseMirror keeps its selection across a blur, so restoring the editor's focus restores the selection with it, and the popup holds no selection state of its own.
 
-Syntax highlighting uses bundled Shiki assets through Leafdown's code highlight plugin. A document's code blocks are all highlighted when it opens; afterwards each transaction maps the existing decorations and highlights again only the code blocks it changed, plus those inside a node whose own boundary it changed, because rebuilding the whole set costs its size times the document's top-level blocks. Code line numbers, while shown, are kept up to date the same way, and turning them on numbers every block. A raw HTML NodeView parses `value` into a source-located inert tree, validates the complete tree against the attribute-free allowlist and self-containment predicate, then constructs the corresponding DOM with element and text-node primitives. The retained source locations map pointer positions in that DOM back to exact offsets in the authored token without reparsing a string into executable markup. A rejected tree renders as muted source text. Rendering never rewrites `value`, and the schema's source-based serialization and clipboard representation remain unchanged.
+Syntax highlighting uses bundled Shiki assets through Leafdown's code highlight plugin. Shiki's JavaScript regular-expression engine takes about half a millisecond per line, so creating the editor state highlights only the document's first 100 lines of code, and the plugin's view highlights the remaining blocks in timer tasks of about 10 ms, each a meta-only transaction outside history, starting from the block at the top of the viewport. Idle callbacks are not used because WebView2 can stop running them for a window entirely. The pending blocks are positions mapped through every transaction; a block whose opening token a transaction deletes or replaces leaves the list, since that transaction highlights whatever replaces it, and a pending block an edit already highlighted is highlighted again in its turn, replacing its decorations. Each transaction maps the existing decorations and highlights again only the code blocks it changed, plus those inside a node whose own boundary it changed, because rebuilding the whole set costs its size times the document's top-level blocks. Code line numbers, while shown, are kept up to date the same way, and turning them on numbers every block. A raw HTML NodeView parses `value` into a source-located inert tree, validates the complete tree against the attribute-free allowlist and self-containment predicate, then constructs the corresponding DOM with element and text-node primitives. The retained source locations map pointer positions in that DOM back to exact offsets in the authored token without reparsing a string into executable markup. A rejected tree renders as muted source text. Rendering never rewrites `value`, and the schema's source-based serialization and clipboard representation remain unchanged.
 
 A math NodeView renders its atom's TeX through the one KaTeX configuration in the math render module, which builds DOM directly and never passes an HTML string, and shows the muted source when the TeX does not parse. The atom's flow follows its paragraph: a `$$` span alone in it is the paragraph's display block. The module reads the TeX out of the source, applying a table cell's `\|` unescaping from the atom's position. Every NodeView renders as it is created, since deferring renders was measured to spread the layout they cost rather than save it. The math projection adapter supplies its preview as a rendered variant of the projection preview widget. Each edit draws the widget again, but the session keeps one preview element, and a display block's preview stands after the paragraph holding its source rather than inside it; otherwise every keystroke would lay the whole rendering out again, which for a large expression costs more than rendering it. While renders stay within a frame's budget the preview renders at once, and a costlier one keeps its rendering until typing pauses.
 

docs/specification.md

@@ -190,7 +190,7 @@ The editor is a unified hybrid Markdown surface. Behavior is governed by renderi
 - Ordered lists render with visual continuation.
 - Clicking a task-list checkbox toggles it checked or unchecked.
 - Tables render as editable table blocks. Basic table editing uses visual table interaction; pipe-delimited Markdown is not exposed in the editor surface. A row holding more or fewer cells than the header is read as the columns the header declares, which is what a Markdown reader shows; cells beyond the header are dropped and missing cells are filled at the end of the row. A pasted HTML body cell spanning columns is flattened into its starting column and ordinary empty cells for the columns it covered, because GFM cannot represent the span. A table written with a header and delimiter row and no body rows is kept and rendered as a header-only table.
-- Code blocks render as styled monospace blocks with syntax highlighting when available. Focused code blocks edit code content directly.
+- Code blocks render as styled monospace blocks with syntax highlighting when available. Focused code blocks edit code content directly. A document with much code is shown before all of it is highlighted: the code it opens on is highlighted at once, and the rest follows in the background while the document can already be read and edited, starting from wherever it is scrolled. Code that is edited before its turn is highlighted as it is edited.
 - A fenced code block whose info string starts with the exact language `math` renders its content as display math while the caret is outside it. Either backtick or tilde fences qualify, and words after `math` remain metadata; `Math`, `latex`, and `tex` do not qualify. Clicking rendered math puts the caret in the code. While the caret is inside, the code stays directly editable with a rendered preview below it. TeX that does not render stays visible as code, with the error available as its description and below the code while editing. Rendering does not change the fence, info string, or content that the block writes to Markdown.
 - A fenced code block whose first info-string word is `mermaid`, ignoring case, renders as a diagram while neither end of the selection is in its code. Other info words are preserved but do not configure the diagram; aliases and indented code do not qualify. Clicking the diagram puts the caret at the start of its code, and an arrow key leaving the neighboring block toward the diagram enters its code at the near edge. While the caret is inside, the code stays directly editable with Mermaid syntax highlighting and the last rendered diagram below it, rendered again once typing pauses. Selecting the whole block shows its diagram. A failed diagram stays editable as source with its error; while the caret is inside, the last rendered diagram stays below the error. Editing the source tries again. Source over 10,000 characters and Mermaid configuration directives or frontmatter stay as source with an explanation. The renderer limits a diagram to 200 edges. A diagram is drawn at its own size in the app's light or dark colors; a diagram wider than the page scrolls horizontally at that size, and pressing its scrollbar does not enter source editing. It is drawn again when the theme changes. The diagram image has a name and a source-text alternative. The authored Markdown, not the image, supplies Undo, Redo, clipboard, block selection and movement, and save. A clean open/save preserves its fence, info string, whitespace, line endings, comments, and directives.
 - A fenced code block names its language on a badge at its top right. Clicking the badge opens the language picker for that block; `Code block language...` opens the same picker from the menu, the context popup, or the keyboard. A block without a language shows a faint `Language` badge only while the pointer is over it or the caret or selection is in it, and takes no extra room for it. Indented code has no fence line to hold a language, so it shows no badge and does not offer the command. The badge is chrome outside the code: it never reaches the saved Markdown, copied content, or counts, and pressing it leaves the selection where it was.
@@ -659,6 +659,7 @@ File operations govern how Leafdown writes to disk and resolves conflicts or err
 
 - Empty files open as empty editable documents.
 - Files larger than 5 MB do not load.
+- Documents up to 200 KB are supported: they are shown within 3 seconds of opening, and each typed character appears within 100 ms without a pause after it. Larger documents up to the 5 MB limit open and can be edited, but opening and typing slow down as they grow, sooner for documents made mostly of tables or code.
 - A file that starts with a UTF-8, UTF-16 LE, or UTF-16 BE byte order mark (BOM) opens in the encoding the BOM names. Any other file opens as UTF-8 unless the user chooses another encoding. Leafdown does not guess an encoding.
 - `Reopen with encoding` is a menu of encodings; choosing one reopens a file without a BOM in that encoding: UTF-8, Windows-1250 to Windows-1258, ISO-8859-2, ISO-8859-15, KOI8-R, KOI8-U, Shift_JIS, EUC-JP, GBK, GB18030, Big5, EUC-KR, or UTF-16 LE or UTF-16 BE without a BOM. ISO-8859-1 is offered as part of the Windows-1252 choice, because the Encoding Standard Leafdown follows decodes the two identically. The invalid encoding error offers the menu from its notification, which stays until the user chooses or dismisses it. The Edit menu and the status bar offer it as a submenu for an open saved document whose file has no BOM, marking the encoding the file was reopened in. A long menu scrolls. Reopening reads the file again into the tab already holding it, and reopening an edited document asks before discarding its changes. A BOM outranks a choice, so a file with one always opens in the encoding its BOM names. The choice applies to that document only and is not remembered, so reopening the file later, including from recent files, reads it as UTF-8 again; the user can choose again whenever the text looks wrong.
 - A file opens in a chosen encoding only when encoding its decoded text again reproduces its bytes exactly. Several multibyte encodings read more than one byte sequence as the same character, so a save of such a file would rewrite bytes the user never edited; Leafdown instead shows an error saying the file cannot be preserved in that encoding, and offers choosing again.

src/features/editor/plugins/codeHighlight.test.ts

@@ -1,25 +1,29 @@
+// @vitest-environment happy-dom
+
 import { Schema, type Node as ProseMirrorNode } from "@milkdown/kit/prose/model";
 import { EditorState, type Transaction } from "@milkdown/kit/prose/state";
 import { liftTarget } from "@milkdown/kit/prose/transform";
-import type { DecorationSet } from "@milkdown/kit/prose/view";
-import { beforeAll, describe, expect, it, vi } from "vitest";
+import { EditorView, type DecorationSet } from "@milkdown/kit/prose/view";
+import { afterEach, beforeAll, describe, expect, it, vi } from "vitest";
 
+import { createCodeBlockDecorations } from "../utils/codeBlockDecorations";
 import { createLeafdownHighlightParser, type LeafdownHighlightParser } from "../utils/highlighting";
 import { createCodeHighlightPlugin, leafdownCodeHighlightPluginKey } from "./codeHighlight";
 
 const schema = new Schema({
   nodes: {
     doc: { content: "block+" },
     text: { group: "inline" },
-    paragraph: { content: "inline*", group: "block" },
+    paragraph: { content: "inline*", group: "block", toDOM: () => ["p", 0] },
     code_block: {
       content: "text*",
       group: "block",
       code: true,
       marks: "",
       attrs: { language: { default: "" } },
+      toDOM: () => ["pre", ["code", 0]],
     },
-    blockquote: { content: "block+", group: "block" },
+    blockquote: { content: "block+", group: "block", toDOM: () => ["blockquote", 0] },
   },
 });
 
@@ -89,14 +93,19 @@ const describeDecorations = (decorations: DecorationSet | undefined) =>
     .toSorted();
 
 const rebuildDecorations = (doc: ProseMirrorNode) =>
-  leafdownCodeHighlightPluginKey.getState(
-    EditorState.create({ doc, plugins: [createCodeHighlightPlugin(shiki)] }),
+  createCodeBlockDecorations(doc, (node, position) =>
+    shiki({
+      content: node.textContent,
+      language: (node.attrs.language as string) || undefined,
+      pos: position,
+      size: node.nodeSize,
+    }),
   );
 
 const applyAndCompare = (state: EditorState, transaction: Transaction) => {
   const next = state.apply(transaction);
 
-  expect(describeDecorations(leafdownCodeHighlightPluginKey.getState(next))).toEqual(
+  expect(describeDecorations(leafdownCodeHighlightPluginKey.getState(next)?.decorations)).toEqual(
     describeDecorations(rebuildDecorations(next.doc)),
   );
 
@@ -117,9 +126,9 @@ describe("code highlighting", () => {
     expect(highlightedPositions(parser)).toEqual(
       findPositions(state.doc, (node) => node.type.name === "code_block"),
     );
-    expect(describeDecorations(leafdownCodeHighlightPluginKey.getState(state))).toContainEqual(
-      expect.stringContaining("--shiki-light"),
-    );
+    expect(
+      describeDecorations(leafdownCodeHighlightPluginKey.getState(state)?.decorations),
+    ).toContainEqual(expect.stringContaining("--shiki-light"));
   });
 
   it("highlights nothing for an edit outside code blocks", () => {
@@ -292,3 +301,118 @@ describe("code highlighting", () => {
     expect(parser).not.toHaveBeenCalled();
   });
 });
+
+const longCode = (language: string, lines: number) =>
+  code(
+    language,
+    Array.from({ length: lines }, (_, line) => `const line${line} = ${line};`).join("\n"),
+  );
+
+const createLongDoc = () =>
+  schema.nodes.doc.create(null, [
+    paragraph("Intro"),
+    longCode("ts", 120),
+    paragraph("Middle"),
+    code("rust", "fn main() {\n    let value = 2;\n}"),
+    quote(code("bash", 'echo "quoted"'), paragraph("Quoted")),
+    code("", "plain text"),
+    code("json", '{ "last": true }'),
+    code("ts", "let edited = 1;"),
+    paragraph("End"),
+  ]);
+
+const mountLongDoc = () => {
+  const parser = vi.fn<LeafdownHighlightParser>((options) => shiki(options));
+  const view = new EditorView(document.createElement("div"), {
+    state: EditorState.create({
+      doc: createLongDoc(),
+      plugins: [createCodeHighlightPlugin(parser)],
+    }),
+  });
+
+  views.push(view);
+
+  return { parser, view };
+};
+
+const views: EditorView[] = [];
+
+const pendingPositions = (state: EditorState) =>
+  leafdownCodeHighlightPluginKey.getState(state)?.pending;
+
+const expectFullyHighlighted = (state: EditorState) => {
+  expect(pendingPositions(state)).toEqual([]);
+  expect(describeDecorations(leafdownCodeHighlightPluginKey.getState(state)?.decorations)).toEqual(
+    describeDecorations(rebuildDecorations(state.doc)),
+  );
+};
+
+describe("code highlighting after the document is shown", () => {
+  afterEach(() => {
+    views.splice(0).forEach((view) => view.destroy());
+    vi.useRealTimers();
+  });
+
+  it("highlights the first lines at once and the remaining blocks once the editor is idle", () => {
+    vi.useFakeTimers();
+    const { parser, view } = mountLongDoc();
+    const blocks = findPositions(view.state.doc, (node) => node.type.name === "code_block");
+
+    expect(highlightedPositions(parser)).toEqual([blocks[0]]);
+    expect(pendingPositions(view.state)).toEqual(blocks.slice(1));
+
+    parser.mockClear();
+    vi.runAllTimers();
+
+    expect(highlightedPositions(parser)).toEqual(blocks.slice(1));
+    expectFullyHighlighted(view.state);
+  });
+
+  it("highlights blocks edited, retyped, added, moved, or removed while others are pending", () => {
+    vi.useFakeTimers();
+    const { view } = mountLongDoc();
+    const at = (index: number) => codeBlockAt(view.state.doc, index);
+
+    view.dispatch(view.state.tr.insertText("    let added = 3;\n", at(1) + 13));
+    view.dispatch(view.state.tr.setNodeAttribute(at(3), "language", "typescript"));
+    view.dispatch(view.state.tr.setNodeMarkup(at(4), undefined, { language: "rust" }));
+    view.dispatch(view.state.tr.insert(at(1), code("bash", "ls -la")));
+    view.dispatch(view.state.tr.delete(at(5), at(5) + view.state.doc.nodeAt(at(5))!.nodeSize));
+    const wrapped = view.state.doc
+      .resolve(at(1) + 1)
+      .blockRange(view.state.doc.resolve(at(2) + 1))!;
+    view.dispatch(view.state.tr.wrap(wrapped, [{ type: schema.nodes.blockquote }]));
+    view.dispatch(view.state.tr.insert(at(3), paragraph("Before")));
+    view.dispatch(view.state.tr.setBlockType(at(5) + 1, at(5) + 1, schema.nodes.paragraph));
+
+    expect(pendingPositions(view.state)!.length).toBeGreaterThan(0);
+
+    vi.runAllTimers();
+
+    expectFullyHighlighted(view.state);
+  });
+
+  it("highlights nothing more once the editor is destroyed", () => {
+    vi.useFakeTimers();
+    const { parser, view } = mountLongDoc();
+
+    parser.mockClear();
+    view.destroy();
+    vi.runAllTimers();
+
+    expect(parser).not.toHaveBeenCalled();
+  });
+
+  it("starts from the block at the top of the viewport", () => {
+    vi.useFakeTimers();
+    const { parser, view } = mountLongDoc();
+    const blocks = findPositions(view.state.doc, (node) => node.type.name === "code_block");
+
+    vi.spyOn(view.dom, "getBoundingClientRect").mockReturnValue(new DOMRect(0, -400, 800, 2000));
+    vi.spyOn(view, "posAtCoords").mockReturnValue({ pos: blocks[3] + 2, inside: -1 });
+    parser.mockClear();
+    vi.runAllTimers();
+
+    expect(highlightedPositions(parser)).toEqual([...blocks.slice(3), blocks[1], blocks[2]]);
+  });
+});