Skip to content
sema-lispPublic

About

Self-hostable package registry for Sema

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

123 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Sema

sema-pkg

Package registry for Sema — a Lisp with first-class LLM primitives.

CI License Website

Self-hostable package registry for the Sema programming language. Ships as a single binary with SQLite, serves both a web UI and a REST API for CLI clients.

Quick Start

jake dev          # start locally (cargo) on a fresh DB and seed it
jake docker   # build + start in Docker on a fresh DB and seed it

Both start the registry on http://localhost:3000 and load the demo data (see Local Development below). If port 3000 is busy they bump to the next free port (and point the seed at the same one) — the chosen URL is printed at startup. Override with jake dev port=4000. jake dev runs the server in the foreground (Ctrl-C to stop); jake docker runs it detached and tails the logs (jake down to stop the container).

To run the server without seeding:

jake run                 # locally, no reset/seed
docker compose up --build

Run jake -l to list all targets.

Local Development

jake dev / jake docker seed a fresh database, then start the server on it. The seeder (src/bin/seed.rs) writes straight to DATABASE_URL through the app's own entities, so it's engine-portable (SQLite/Postgres/MySQL), fast, and needs no running server. Seeded users have real password hashes (they can log in), featured packages have real downloadable blobs, and it prints a working API token.

The reproducible demo dataset:

  • Users: helge (admin), kari, magnus, ingrid, olav, spambot (banned), plus realistic bulk accounts. Every seeded user has the password 123123123. Admin login: helge / 123123123, panel at /admin.
  • Packages: featured sema-http, sema-json, sema-router, sema-sql, sema-cli, sema-async, sema-test, sema-crypto, sema-csv, sema-log (multiple versions, some yanked, READMEs, real blobs) plus dozens of realistic packages.
  • Downloads, reports, audit log: populated with a believable distribution.
jake seed                       # small realistic dev set (wipes + reseeds)
jake seed-stress                # bulk: ~1k users, ~500 packages
# Any engine — just point DATABASE_URL at it:
DATABASE_URL="postgres://sema:sema@localhost:5432/sema" jake seed
# Or run the binary directly for flags (--fresh, --large):
cargo run --features seed --bin seed -- --fresh --large

For scale testing, --huge seeds 20k users and 1M packages (~4.5M versions) in under a minute on SQLite; SEED_USERS / SEED_PACKAGES override any preset.

Package API references

Uploaded releases get an API reference at /packages/{name}/{version}/api. The package page links to it. The reference includes a searchable export index, signatures with distinct argument styling, syntax-highlighted definitions, and links to the stored source with line numbers. Two other formats use the same stored document:

  • Append .md to the reference URL for Markdown (text/markdown).
  • Append /print for a standalone HTML page with inline styles and no external assets. Save that page for offline reading or print it from the browser.

Extraction uses the pinned Sema reader, without running package code. It reads the explicit [package].entrypoint in sema.toml, literal top-level modules and their export lists, direct function declarations, and leading string docstrings with a following function body. It does not invent parameter types, examples, or a documentation syntax. Aliases, computed definitions, and ambiguous bindings are shown as unresolved. Unsupported source produces an explanation and a README link; it does not prevent publication.

Each document records the archive checksum, source checksum, and parser version. The parser accepts at most 16 MiB of expanded archive data, 4,096 archive entries, 512 KiB per source file, and 512 exports. Documents are stored once per release. GitHub-linked releases without a registry-stored archive cannot be documented this way; the registry does not fetch mutable repository files to fill gaps.

Backfill existing stored uploads after deploying the migration:

sema-pkg package backfill-api-docs

The command verifies each stored archive's SHA-256, skips existing documents and GitHub-linked releases, and processes versions in bounded batches. Set API_DOCS_ENABLED=false to disable generation, links, routes, and backfill.

For a local preview, start the registry with a separate database and blob directory on port 4177, then run:

node scripts/preview-api-docs.mjs http://127.0.0.1:4177

The script downloads checksum-verified public releases of sema-retry, sema-cache, and sema-medley, and publishes them to the local server through its normal API. It accepts only loopback targets and prints the reference URLs.

Observability

Traces (OpenTelemetry) and metrics (Prometheus) are built in and off until configured — nothing is exported by default. See DEPLOYMENT.md for a full local Jaeger + Prometheus

  • Grafana stack.

Traces — pick an exporter with OTEL_TRACES_EXPORTER:

jake trace                                   # → traces.jsonl (one span per line)
OTEL_TRACES_EXPORTER=otlp \
  OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 sema-pkg   # → Jaeger/Tempo

Requests, handlers, and DAL queries nest into a request → handler → query tree; each span carries duration_ms and the source code.file.path/line. For the file exporter, render the slowest spans with jq:

jq -r '[.name,(.duration_ms|tostring)+"ms",(.attributes.op//"")]|@tsv' traces.jsonl \
  | sort -t$'\t' -k2 -rn | column -t -s$'\t'

Metrics — METRICS_ENABLED=true serves Prometheus RED metrics at /metrics (http_requests_total, http_request_duration_seconds, http_requests_in_flight), labelled by matched route.

RUST_LOG sets log verbosity; OTEL_LOG sets span capture independently, so quiet logs and rich traces coexist.

Configuration

All configuration is via environment variables with sensible defaults:

Variable Default Description
HOST 0.0.0.0 Bind address
PORT 3000 Listen port
DATABASE_URL sqlite://data/registry.db?mode=rwc Database URL. The engine is inferred from the scheme — sqlite:, postgres:, or mysql: (see Database engines).
DATABASE_MAX_CONNECTIONS 10 Connection pool size (read concurrency).
BLOB_DIR data/blobs Directory for package tarballs (used when S3 is not configured)
BLOB_S3_BUCKET — Bucket for S3-compatible blob storage (Cloudflare R2, MinIO, S3). When set, tarballs go to object storage instead of BLOB_DIR — required for stateless / multi-instance deploys.
BLOB_S3_ENDPOINT — S3 endpoint URL (e.g. https://<account>.r2.cloudflarestorage.com). Omit for AWS S3.
BLOB_S3_REGION auto Region. auto for R2.
BLOB_S3_ACCESS_KEY_ID — Access key (required when BLOB_S3_BUCKET is set)
BLOB_S3_SECRET_ACCESS_KEY — Secret key (required when BLOB_S3_BUCKET is set)
BASE_URL http://localhost:3000 Public URL (used in links; enables Secure session cookies when https://)
MAX_TARBALL_BYTES 52428800 (50 MB) Max upload size
MAX_DEPENDENCIES 64 Max dependencies per published version
API_DOCS_ENABLED true Generate and serve API references for stored releases; also controls backfill.
GITHUB_CLIENT_ID — GitHub OAuth app client ID (optional)
GITHUB_CLIENT_SECRET — GitHub OAuth app secret (optional)
OAUTH_TOKEN_KEY — 32-byte key encrypting stored GitHub tokens. Required when GitHub OAuth is enabled — the server refuses to boot if left at the insecure default.
RATE_LIMIT_ENABLED true IP-keyed request rate limiting (see DEPLOYMENT.md). Disable only behind a gateway that limits for you.
RATE_LIMIT_RPS 20 Sustained requests/sec per IP on the general/write API
RATE_LIMIT_BURST 40 Burst allowance per IP before throttling
RATE_LIMIT_READ_RPS 100 Sustained requests/sec per IP on the install path (metadata + download); generous so multi-package installs aren't throttled
RATE_LIMIT_READ_BURST 500 Burst allowance per IP on the install path

Database engines

The registry runs on SQLite, PostgreSQL, or MySQL from the same binary — the engine is chosen by the DATABASE_URL scheme:

DATABASE_URL="sqlite://data/registry.db?mode=rwc"          # default
DATABASE_URL="postgres://user:pass@host:5432/sema"         # PostgreSQL
DATABASE_URL="mysql://user:pass@host:3306/sema"            # MySQL

On startup the server infers the backend, applies SQLite-only tuning (WAL) where relevant, and runs the schema migrations (src/migration/, SeaORM programmatic migrations that emit correct DDL per engine). No manual schema step is needed.

After deploying the release-documentation migration, run this once against the production database and its original upload blob store to capture verified historical release READMEs and changelogs:

sema-pkg package backfill-readmes

It processes archive metadata in bounded batches, checks each archive's stored SHA-256, and skips GitHub-linked releases because their original archives are not stored by the registry.

All database access goes through the Data Access Layer in src/dal/ (one module per aggregate: packages, versions, owners, deps, users, sessions, tokens, reports, audit_log, oauth, downloads, plus admin read models and a time helper). Handlers never touch SQL directly, and the DAL avoids engine-specific constructs — timestamps are generated in Rust, upserts use SeaORM's on_conflict, and any raw SQL is standard and parameterized. This is what keeps all three engines behaving identically; add new queries to the DAL, not to handlers.

Run the suite against all three engines (Docker):

jake all-drivers   # SQLite + PostgreSQL + MySQL via docker-compose.test.yml

API Endpoints

Auth

Method Path Auth Description
POST /api/v1/auth/register — Create account {username, email, password}
POST /api/v1/auth/login — Sign in {username, password}
POST /api/v1/auth/logout — Clear session

Tokens

Method Path Auth Description
POST /api/v1/tokens Session Create API token {name}
GET /api/v1/tokens Session List your tokens
DELETE /api/v1/tokens/{id} Session Revoke a token

Packages

Method Path Auth Description
GET /api/v1/search?q=&page=&per_page= — Search packages
GET /api/v1/packages/{name} — Package metadata + versions
PUT /api/v1/packages/{name}/{version} Bearer Publish version (multipart: tarball + metadata)
GET /api/v1/packages/{name}/{version}/download — Download tarball
POST /api/v1/packages/{name}/{version}/yank Bearer Yank a version

Ownership

Method Path Auth Description
GET /api/v1/packages/{name}/owners — List owners
PUT /api/v1/packages/{name}/owners Bearer Add owner {username}
DELETE /api/v1/packages/{name}/owners Bearer Remove owner {username}

GitHub-Linked Packages

Method Path Auth Description
POST /api/v1/packages/link Session Link a GitHub repo {repo_url}
POST /api/v1/packages/{name}/sync Session Manual re-sync from GitHub (owner only)
POST /api/v1/webhooks/github HMAC Webhook receiver for tag events

GitHub-Linked Packages

Link a GitHub repository to automatically publish packages from semver tags.

Prerequisites

  • GitHub OAuth configured (GITHUB_CLIENT_ID, GITHUB_CLIENT_SECRET)
  • OAUTH_TOKEN_KEY set to a random 32-character string (used to encrypt stored GitHub tokens)

How It Works

  1. User connects their GitHub account via OAuth
  2. User pastes a repo URL on the /link page
  3. Registry validates the repo contains a sema.toml, then imports existing semver tags as versions
  4. A webhook is registered on the repo — new semver tags are published automatically

Tag-to-Version Mapping

Git tags are mapped to package versions: v1.0.0 → 1.0.0. Tags that don't match semver (e.g., nightly, latest) are skipped.

Sema Version Compatibility

Packages may declare a SemVer requirement in sema.toml:

[package]
sema_version_req = ">=1.34.0"

The registry validates and normalizes this field for both CLI uploads and GitHub-linked releases. Clients use it to skip incompatible versions during unversioned installs and to reject incompatible explicit or locked installs before replacing local package files.

For a GitHub-linked package the requirement is read from sema.toml at each tag, so every release records the requirement it was published with and editing the field does not rewrite past releases. A tag whose sema.toml is missing or unreadable records no requirement; a tag whose sema.toml parses but holds an invalid requirement is reported in the package's sync log.

Source Locking

A package is either CLI-uploaded or GitHub-linked, never both. Once a package is linked to a repo, it cannot be published via sema publish, and vice versa.

Self-Hosting

  1. Build: cargo build --release
  2. Copy target/release/sema-pkg, templates/, and static/ to your server (schema migrations are compiled into the binary and run automatically on startup)
  3. Set DATABASE_URL, BLOB_DIR, BASE_URL (use an https:// URL so session cookies are marked Secure), and — if using GitHub OAuth — a unique OAUTH_TOKEN_KEY
  4. Run sema-pkg behind a reverse proxy (nginx/caddy) with TLS

Or use the Docker image:

docker compose up -d

Data is stored in ./data/ (SQLite DB + blob files). Back up this directory.

Operate the registry from the CLI (works on any backend, no manual DB edits):

sema-pkg admin create <user> <email> <pw>   # create the first admin (API can't)
sema-pkg admin promote|demote|ban|unban <user>
sema-pkg admin reset-password <user> <pw>   # recover a locked-out account
sema-pkg admin token-create <user> [name]   # mint an API token (e.g. for a bot account)
sema-pkg admin revoke-tokens <user>         # after a token leak
sema-pkg package yank <name> <version>      # moderation / takedowns
sema-pkg package remove <name>
sema-pkg stats                              # user/package/download counts
sema-pkg doctor                             # check DB, blob store, config

For production hosting — Fly.io (recommended: SQLite + Tigris + built-in Litestream backup), Docker Compose, a bare binary + systemd, or managed Postgres for horizontal scale — see DEPLOYMENT.md.

GitHub OAuth (Optional)

  1. Create a GitHub OAuth App at https://github.com/settings/developers
  2. Set callback URL to {BASE_URL}/auth/github/callback
  3. Set GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET environment variables
  4. Restart the server — the GitHub login button will appear automatically

License

MIT

About

Self-hostable package registry for Sema

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages