HeroParser - High-Performance, AI-Native CSV, Fixed-Width, Excel (.xlsx), JSONL & HTB Parser

HeroParser is a low-allocation, SIMD-accelerated tabular data parser and writer for .NET 8, 9, and 10. Designed for extreme speed, memory efficiency, and Native AOT compatibility, it also offers first-class integrations for AI agents, vector embeddings, and LLM pipelines.

Why Choose HeroParser?

• Extreme Performance: Engineered with AVX-512, AVX2, and ARM NEON SIMD optimizations to deliver ultra-high-throughput reading and writing.
• AI-Native integrations: Built-in support for token-budgeted chunking, LLM output structured repair, vector embedding pipelines, and agent tool mapping.
• Low Footprint: Span row readers expose fields without creating a string for every field. Typed records, decoded strings, and streaming APIs have their own allocation costs; see the benchmark scope and evidence.
• Unified Attributes: Annotate your C# classes once, and use them across CSV, Excel, Fixed-Width, and HTB APIs.

---

Performance & Memory

Performance depends on the input, API, runtime, and CPU. Both UTF-8 (byte[]) and UTF-16 (string) span readers support SIMD scanning; use the representation you already have to avoid unnecessary encoding conversions.

The 2026-10-23 CI report, Base cba2bcd measured 176 B per operation for HeroParser's legacy 10,000-row × 25-column count-only fixture on AMD EPYC 7763, Ubuntu 24.04, .NET 10.0.13. Input preparation was outside the timed operation; the test did not decode every field, bind typed records, or perform file I/O. Its library options were not equivalent, so the archived comparison does not establish a general performance ranking. The corrected fixture requires a new baseline.

See the Benchmarks Guide for the historical table, limitations, current dependencies, and commands for measuring your own workload. Capacity estimates in text writers can reduce buffer growth, but output strings and application objects still allocate.

View live performance graphs and history on the HeroParser Performance Portal.

---

Install

Core Library

dotnet add package HeroParser

High-Performance Console Engine

dotnet add package HeroParser.Console

Command-Line Utility (CLI)

Option 1: Dotnet Global Tool (Cross-Platform)

dotnet tool install --global HeroParser.Cli

Option 2: Homebrew Tap (macOS & Linux)

Install the native binary using Homebrew:

brew tap KoalaFacts/heroparser
brew install heroparser

Option 3: Shell Installer Script (macOS & Linux fallback)

Install the native binary without Homebrew:

curl -fsSL https://raw.githubusercontent.com/KoalaFacts/HeroParser/main/install.sh | sh

Option 4: Snap Store (Linux, pending)

The Snap package is not yet publicly available in the Snap Store. Use the shell installer above or a Linux archive from GitHub Releases until store publication is confirmed.

Option 5: WinGet (Windows)

Install the portable binary via WinGet:

winget install KoalaFacts.HeroParser

Option 6: Scoop (Windows)

Install via Scoop:

scoop bucket add heroparser https://github.com/KoalaFacts/scoop-heroparser.git
scoop install heroparser/heroparser

WebAssembly / JavaScript (Node.js & Browser)

The JavaScript package is an unpublished preview in this repository, not yet available from npm. The workspace requires npm 12 or newer and rejects unreviewed dependency install scripts. Build the WebAssembly runtime locally before using the workspace package; see the package README. The browser demo is separate from an npm release.

Interactive Playground Demo

Try the interactive WebAssembly sandbox directly in your browser: 👉 https://KoalaFacts.github.io/HeroParser/demo/

JS/TS Quick Start

import { init, readCsv } from 'heroparser';

// Initialize the WebAssembly runtime
await init();

// Parse CSV text directly in the browser or Node.js
const records = readCsv("Name,Age,Role\nAlice,30,Developer\nBob,25,Designer", {
    delimiter: ',',
    hasHeader: true
});
console.log(records);

---

Simple Quick Starts

Define your record type. Decorate it with [GenerateBinder] to enable source-generated, reflection-free, and Native AOT-safe binding:

using HeroParser;

[GenerateBinder]
public class Product
{
    public int Id { get; set; }
    public string Name { get; set; } = "";
    
    [Validate(RangeMin = 0)]
    public decimal Price { get; set; }
    
    [Parse(Format = "yyyy-MM-dd")]
    public DateTime ReleaseDate { get; set; }
}

CSV

// Read a file into typed records
List<Product> products = Csv.Read<Product>().FromFile("products.csv").ToList();

// Async stream millions of rows without buffering
await foreach (Product p in Csv.Read<Product>().FromFileAsync("products.csv"))
{
    Console.WriteLine($"{p.Name}: {p.Price:C}");
}

// Write collection to a file
Csv.Write<Product>().ToFile("out.csv", products);

Excel (.xlsx)

Reads and writes Excel workbooks with zero external dependencies (utilizes only standard .NET compression and XML packages).

using HeroParser.Excels.Core;

// Read Excel files
List<Product> products = Excel.Read<Product>().FromFile("products.xlsx");

// Write Excel workbook with custom header styles, column styles, and auto-merged duplicate values
var headerStyle = ExcelStyle.Create()
    .WithFont(f => f.WithName("Arial").WithSize(12).WithBold().WithColor("FFFFFF"))
    .WithFill(fill => fill.WithSolidColor("007ACC")); // Blue background

Excel.Write<Product>()
    .WithHeaderStyle(headerStyle)
    .WithMergeDuplicates(p => p.Name) // Vertically merge contiguous duplicate Names
    .ToFile("out.xlsx", products);

Fixed-Width

Map properties to specific character boundaries using the [PositionalMap] attribute:

[GenerateBinder]
public class Employee
{
    [PositionalMap(Start = 0, Length = 10)]
    public string Id { get; set; } = "";

    [PositionalMap(Start = 10, Length = 30)]
    public string Name { get; set; } = "";

    [PositionalMap(Start = 40, Length = 10, Alignment = FieldAlignment.Right)]
    public decimal Salary { get; set; }
}

// Read positional records
var employees = FixedWidth.Read<Employee>().FromFile("employees.dat").Records;

// Write positional records
FixedWidth.Write<Employee>().ToFile("out.dat", employees);

JSON Lines (JSONL)

Perfect for AI fine-tuning datasets, streamed LLM responses, and database bulk loading.

using System.Text.Json.Serialization;

// Read JSONL files using source-generated JSON metadata (AOT-safe)
using var reader = Jsonl.Read<Product>()
    .FromFile("products.jsonl", ProductJsonContext.Default.Product);
var records = reader.ToList();

// Write JSONL files
Jsonl.Write<Product>().ToFile("out.jsonl", records, ProductJsonContext.Default.Product);

[JsonSerializable(typeof(Product))]
internal partial class ProductJsonContext : JsonSerializerContext { }

JSONL uses the System.Text.Json source-generated context above; [GenerateBinder] does not generate JSON serialization metadata.

High-Throughput Tabular Binary (HTB)

A custom binary serialization format with source-generated mapping, platform-independent byte order, and vector embedding storage (supporting float[] arrays).

Why HTB?

• Source-Generated Mapping: Uses Roslyn source generators ([GenerateBinder]) to map supported properties without reflection. Records, strings, and arrays still have allocation costs.
• Vector Embedding Native: Natively supports floating-point arrays (float[]), enabling ultra-fast vector embedding serialization without string parsing overhead.
• Platform-Independent Endianness: Automatically handles big-endian byte-order reversal for floats, doubles, and ints for cross-architecture safety.
• CSV ↔ HTB Conversion: Provides synchronous file conversion and asynchronous stream conversion between CSV and HTB.
• AOT & Trim Ready: Fully compatible with Native AOT compilation out-of-the-box.

// Read HTB binary files (AOT-safe)
List<Product> products = Htb.Read<Product>().FromFile("products.htb").ToList();

// Async stream HTB records
await foreach (Product p in Htb.Read<Product>().FromFileAsync("products.htb"))
{
    Console.WriteLine($"{p.Name}: {p.Price:C}");
}

// Write records to an HTB file
Htb.Write<Product>().ToFile("out.htb", products);

// Direct CSV ↔ HTB conversion
Htb.ConvertFromCsv("products.csv", "products.htb", HtbSchema.FromType<Product>());

Console (High-Performance Terminal Widget Engine)

An allocation-conscious, reflection-free, and Native AOT-compatible library designed for high-performance terminal applications.

using HeroParser.Console;

// Render styled ANSI markup
AnsiConsole.MarkupLine("[bold green]Success:[/] Row validation completed in [yellow]4.2ms[/].");

// Render styled tables and panels
var table = new Table().Border(TableBorder.Rounded);
table.AddColumn("[blue]Filename[/]");
table.AddColumn("[blue]Records[/]");

table.AddRow("data.csv", "10,240");
table.AddRow("data.jsonl", "102,400");

AnsiConsole.Write(table);

---

Unified Attribute System

Annotate a single record class once, and read or write it across multiple formats:

Attribute	Purpose	CSV	Excel	Fixed-Width	HTB
[GenerateBinder]	Emits Roslyn source-generated, reflection-free mapping binder	Yes	Yes	Yes	Yes
[TabularMap(Name, Index)]	Maps property to column header or index	Yes	Yes	No	Yes
[PositionalMap(Start, Length...)]	Declares character position, alignment, and pad characters	No	No	Yes	No
[Parse(Format)]	Converts raw values to custom types (e.g. DateTime format)	Yes	Yes	Yes	No
[Format(WriteFormat...)]	Customizes output formatting during serialization	Yes	Yes	Yes	No
[Validate(Range, Pattern...)]	Validates properties bidirectionally (Strict/Lenient modes)	Yes	Yes	Yes	Yes

---

AI-Native Tabular Capabilities

HeroParser includes first-class support for LLM, vector search, and RAG pipelines:

1. LLM Structured Output Repair (LlmRepair)

Repairs truncated final rows (unclosed quotes/escapes) and strips markdown code blocks from raw LLM text streams.

using HeroParser.AI;

// Repaired on-the-fly and parsed directly into strongly-typed records
await foreach (var dev in LlmRepair.ReadFromTextAsync<Developer>(rawLlmResponse))
{
    Console.WriteLine($"{dev.Name} is a {dev.Role}");
}

2. Tabular Embedding Pipeline (ToLlmEmbeddingsAsync)

Batches streamed records and pairs them with vector embeddings using a token-budgeted streaming wrapper. Chunks, request text, and returned embedding arrays have allocation costs.

using HeroParser.AI;

await foreach (var chunk in developers.ToLlmEmbeddingsAsync(
    async (texts, ct) => await GetEmbeddingsFromApiAsync(texts, ct),
    options: new LlmChunkOptions { MaxTokensPerChunk = 250 },
    batchSize: 16))
{
    Console.WriteLine($"Chunk of {chunk.Chunk.EndRow - chunk.Chunk.StartRow + 1} rows embedded.");
}

3. Agent Tool Argument Mapper (SchemaMetadata.MapFromToolCall)

Maps flat dictionaries of case-insensitive arguments returned by tool calling models into typed record models, executing validation constraints and raising rich validation feedback.

using HeroParser.AI;

Developer dev = SchemaMetadata.MapFromToolCall<Developer>(arguments);

4. Tabular Context Profiler (TabularContextProfiler)

Generates structured statistical profile cards in markdown directly from datasets to inject into LLM system prompts.

string contextCard = developers.GenerateContextCard(datasetName: "Engineering Team");

5. Token-Bounded JSON Chunker (JsonLlmChunker)

Chunks datasets into valid, token-bounded JSON array blocks, optimized for ingestion into RAG context windows.

await foreach (var jsonChunk in developers.ToJsonLlmChunksAsync(options)) { ... }

---

Key Features

• SIMD-accelerated CSV parsing — AVX-512, AVX2, and ARM NEON instruction sets; PCLMULQDQ-based branchless quote tracking
• Allocation-conscious scanning — span field access and pooled buffers; allocation profiles vary by API, options, and output representation
• AOT/trimming ready — source generators emit reflection-free binders; annotated with [RequiresUnreferencedCode] where reflection is unavoidable
• Async streaming — IAsyncEnumerable<T> for CSV, Fixed-Width, Excel, JSONL, and HTB; true non-blocking I/O with sync fast paths
• Excel without extra dependencies — reads and writes .xlsx using only System.IO.Compression and System.Xml
• JSONL for AI/ML pipelines — Jsonl.Read<T>() / Jsonl.Write<T>() mirror the CSV builder pattern; AOT-safe via JsonTypeInfo<T>; CsvToJsonlConverter projects tabular data into OpenAI/Anthropic fine-tuning shapes
• HTB binary format — float-array embedding support, big-endian byte-order reversal, and direct CSV ↔ HTB conversion
• Embedding-API batching — IAsyncEnumerable<T>.BatchAsync(size) groups streamed records into fixed-size batches for OpenAI/Voyage/Cohere/Anthropic embedding calls
• Inline vector parser — VectorParser.ParseFloats(span) handles pre-computed embeddings ("[0.1,0.2,…]", comma/semicolon/whitespace separators, culture-aware)
• DataReader support — Csv.CreateDataReader(), FixedWidth.CreateDataReader(), Excel.CreateDataReader(), Jsonl.CreateDataReader() for database bulk loading via SqlBulkCopy
• PipeReader integration — Csv.ReadFromPipeReaderAsync(pipe) for network streaming without buffering the entire payload
• Multi-schema CSV — discriminator-based row routing to different record types, with runtime and source-generated dispatch
• Delimiter detection — auto-detect comma, semicolon, pipe, or tab from sample rows with a confidence score
• CSV validation — pre-flight structural checks with detailed per-row error reporting
• Field validation — [Validate] constraints (NotNull, NotEmpty, Range, Pattern) collected lazily; result.ThrowIfAnyError() for fail-fast
• CSV injection protection — configurable sanitization modes for user-data exports
• Progress reporting — row/byte callbacks for large-file UX
• Custom type converters — register converters for domain types on any reader or writer
• Write capacity pre-allocation — estimates backing buffer capacity from collection counts when writing to strings to reduce buffer growth; actual sizes and benefits depend on the records
• Multi-framework — .NET 8, 9, 10; CI validates all three on Windows, Linux, and macOS

---

Detailed Documentation

For advanced features and full API guides, see the files under the docs folder:

• CSV Guide — Fluent readers/writers, validation, PipeReader, and multi-schema dispatching.
• Excel Guide — Multi-sheet workbooks, custom formatting, and progress tracking.
• Fixed-Width Guide — Positional mapping, alignment, padding, and custom type converters.
• JSONL Guide — Fine-tuning templates, vector parsing, and Native AOT setups.
• HTB Guide — High-Throughput Tabular Binary format, fluent APIs, CSV ↔ HTB conversion, and Native AOT support.
• Console Guide — Standard rendering, markup, widgets (Table, Panel, Rule), and interactive prompts.
• CLI Guide — Local-First AI-native architecture, auto-detecting terminal CLIs (agy/openai/claude/copilot/ollama), stdin streaming, process management, and interactive query/translate wizard.
• Benchmarks Guide — Execution environments, detailed CPU metrics, and comparisons.

---

Building & Testing

# Build all projects
dotnet build

# xUnit v3 uses Microsoft.Testing.Platform; pass MTP arguments after `--`.
# Run unit tests
dotnet test tests/HeroParser.Tests -- --filter-trait Category=Unit

# Run integration tests
dotnet test tests/HeroParser.Tests -- --filter-trait Category=Integration

# Run all tests
dotnet test

# Check code formatting
dotnet format --verify-no-changes

# Run benchmarks
dotnet run -c Release --project benchmarks/HeroParser.Benchmarks

---

License

MIT