Bootstrap Academy Challenges Microservice

The official challenges microservice of Bootstrap Academy.

If you would like to submit a bug report or feature request, or are looking for general information about the project or the publicly available instances, please refer to the Bootstrap-Academy repository.

Development Setup

1. Install the Rust stable toolchain.
2. Clone this repository and cd into it.
3. Install Just (cargo install just) and Sea-ORM (cargo install sea-orm-cli)
4. Start a PostgreSQL database, for example using Docker or Podman:

   podman run -d --rm \
       --name postgres \
       -p 127.0.0.1:5432:5432 \
       -e POSTGRES_HOST_AUTH_METHOD=trust \
       postgres:alpine

5. Create the academy-challenges database:

   podman exec postgres \
       psql -U postgres \
       -c 'create database "academy-challenges"'

6. Start a Redis instance, for example using Docker or Podman:

   podman run -d --rm \
       --name redis \
       -p 127.0.0.1:6379:6379 \
       redis:alpine

7. Run just migrate to run the database migrations.
8. Run just run to start the microservice. You can find the automatically generated swagger documentation on http://localhost:8005/docs.

Coding execution

After running migrations, challenges starts the API and its coding worker as before. To scale them separately, run challenges api and one or more challenges worker processes with the same database and service configuration. Workers do not bind an HTTP port. Set challenges.coding_challenges.execution.embedded_worker = false to make the default command serve only the API instead. The existing max_concurrency setting limits execution slots per worker process.

The optional [challenges.coding_challenges.execution] configuration defaults to:

embedded_worker = true
max_pending = 1024
max_pending_per_user = 4
lease_seconds = 30
poll_milliseconds = 500
retry_seconds = 10
max_execution_seconds = 600

Use consistent admission limits across API replicas. Pending includes running and technically deferred submissions; an overflowing queue responds with the existing HTTP 429 retry response before storing a submission or charging a heart. Previously accepted work remains queued even if it exceeds a newly configured limit. Technical executor failures stay pending and retry without charging hearts.

PostgreSQL claims only committed submissions and fences result/progress/outbox commits by lease owner and generation. Heartbeats renew a live lease every third of its duration. The queue API reports live advertised execution capacity, actual leased submissions, and waiting submissions across processes. Position zero means actively leased; positive positions are an approximate waiting order and may change during retries or concurrent work.

Stop or drain all old workers before starting this version's workers: older binaries do not participate in leases. Apply the additive migration before the new binary. During recovery stop all new workers before starting the previous binary and retain the additive schema. Existing submission IDs, results, heart operations and benefit outboxes are preserved. If a process or connection fails, remote sandbox work can physically run again after lease expiry; fencing prevents an obsolete worker from committing results or rewards, but does not promise exactly-once remote computation.

Local database regression (fresh migrated disposable PostgreSQL plus isolated Redis, no live services):

HEART_TEST_DATABASE_URL=postgresql://... HEART_TEST_REDIS_URL=redis://... \
  cargo test --locked -p challenges coding_durable_execution_postgres -- --ignored --test-threads=1

Run that test separately from the historical heart fixtures, which intentionally leave submission history behind. cargo test --locked --workspace runs the regular checks without external fixture dependencies.

For the complete local migration/heart/authority/export regression and two real worker processes against a deliberately stalled local executor, use Python 3.11+, the PostgreSQL/Redis tools on PATH, and cached Cargo dependencies:

python3 scripts/test-coding-execution.py --output /tmp/coding-execution-check

The output directory must be new. The script binds all fixtures to loopback, keeps its logs, and stops/removes only its own processes and database cluster.

Learning access and daily lessons

Attempt admission reads the current Backend policy from GET /shop/_internal/learning-policy/{user_id}. legacy and shadow retain the existing heart rules; daily bypasses both the balance requirement and new wrong-answer heart operations. Existing admin, author and retired-task exceptions remain. Missing users, unknown modes, invalid responses and service failures return an error before accepting an attempt. The policy read never debits; while a confirmed, funded monthly Premium renewal is due, Backend answers it with 500. Admission then reads GET /shop/_internal/premium/{user_id} once, which settles the renewal like the Premium status read, and asks the policy again. Only that policy answer decides; if settlement fails, the attempt stays a retryable 503.

Skills owns course access, lesson binding resolution and the daily counter. Challenges supplies authenticated user IDs, task/subtask IDs and course, section and lecture bindings from its database to POST /skills/_internal/learning-access/{user_id}/check or /start. Individual read/code recovery routes only check access. Quiz answers, coding submissions and running a coding example start learning before accepting work.

Metadata/statistics list admission is opt-in through challenges.learning_access_reads (or CHALLENGES__LEARNING_ACCESS_READS=true), default false. Full-content question, matching, multiple-choice and coding lists always check the concrete course/lesson rights of their detail routes, including with that switch off. Deploy the Skills check-batch route before this Challenges version. Content and active metadata lists load parent bindings once and check batches of at most 250 concrete subtasks through POST /skills/_internal/learning-access/{user_id}/check-batch; concrete IDs preserve lesson-specific rights, including during policy outages. Course-task lists check each parent binding once. These checks never start a lesson. The attempt/submission UUID is the internal request ID; Skills' durable user/course/lesson start makes retrying an uncertain admission safe. Existing public challenge POSTs keep their original attempt semantics.

The same checks apply to raw exercise IDs and retained-learning routes. Inaccessible rows are omitted from lists and details keep their existing 404 envelopes. Start refusals preserve Skills' status and JSON, including 429 daily_limit_reached with the current daily status. Provider errors do not become incorrect answers or grant access. Shared and standalone exercises are resolved by Skills; an inaccessible optional binding must not suppress an otherwise eligible exercise. Course/section-wide bindings keep nullable lecture IDs, so they cannot masquerade as standalone work.

Heart outbox settlement always submits the original operation to Backend. The final daily_learning receipt requires zero charged half-hearts. Already completed operations replay their original receipt, including a charge made before a policy transition. Technical errors and malformed receipts remain pending; accepted learner results and reward idempotency are unchanged.

Deploy the compatible Backend policy/receipt API and Skills check/start API before this consumer. Backend's receipt CHECK migration must precede daily activation. Backend defaults and cohort/terms eligibility control learning policy; Skills independently controls daily measurement/enforcement. This consumer does not activate either policy or introduce new terms or prices.

Skills reads historical participation through the service-local POST /_internal/users/{user_id}/learning-history, authenticated with the existing internal JWT audience challenges (including per-audience secrets). The body is {subtask_ids: UUID[], lecture_bindings: [{course_id, lecture_id}]}; both lists default to empty, and the combined limit is 500 entries before deduplication. Invalid or oversized requests return 422. The response contains only {attempted_subtask_ids: UUID[], attempted_lecture_bindings: [{course_id, lecture_id}]}, with unique results restricted to the requested IDs and exact course/lecture pairs. An unknown user or empty request returns empty lists.

This read-only local query recognizes wrong and solved quiz attempts, submitted code including pending judgments, and saved progress with an attempt count or last-attempt/solved timestamp. Empty or rating-only progress, views and broad course/section bindings do not start a lecture. Historical example executions have no durable user participation record here and cannot be reconstructed from shared evaluator caches. No code, answers, solutions or success claims leave this endpoint, and it does not call Skills or modify XP, receipts or progress. Deploy this endpoint before the Skills consumer that uses its evidence to preserve historical lesson starts. The two services may call each other during admission, but this history lookup has no upstream dependency or subject lock.

The API gives this exact POST route a separate database pool with at most one additional connection per API process. Its read-only snapshot can finish when all ten primary connections are held by attempts waiting for Skills admission. Other routes keep the original request transaction, commit/rollback behavior, and primary pool; the callback keeps the same authentication and batch limits. Workers and deletion sweeps do not create the callback pool.

Run the native access, heart, historical-authority, coding queue and moderation-effect regressions with cached Cargo dependencies and PostgreSQL/Redis tools on PATH:

python3 scripts/test-learning-access.py --output /tmp/learning-access-check

The runner uses fresh disposable databases and local HTTP contract fixtures, records command results, and stops/removes its own services. It supports CARGO_TARGET_DIR. This tests the Challenges boundary; Skills counter/day concurrency and real multi-service release acceptance are separate checks.

Account Deletion

When an account is deleted, the auth microservice calls DELETE /_internal/users/:user_id on this microservice. The endpoint requires an internal token with the challenges audience and answers 204, also for a user that has no data here, so it can be retried safely.

It deletes the bans the user received, their subtask reports, their multiple choice, question and matching attempts, their coding challenge submissions and their user subtask rows, as well as the subtasks they created — including everything referencing those subtasks, which the database removes through ON DELETE CASCADE. Bans the user issued stay in force with the nil uuid as their creator. Tasks are shared between users, so a task the user created is only deleted if no subtask is left in it; otherwise it stays with the nil uuid as its creator, so the user's id is gone from the database afterwards. The records of the user's first completions (challenge_benefit_earnings) go in the same transaction, with their delivery components and observations; a component that has not been delivered yet is dropped, because the deleted account can no longer receive it. The cached values tagged with the user id are dropped afterwards.

Because the auth microservice logs and swallows a failing call, a periodic sweep catches the deletions that were lost:

cargo run -- sweep-deleted-users   # in the dev setup; on hosts with the NixOS module see below

It walks the distinct user ids referenced anywhere in the database in batches, asks the auth microservice for each one and then deletes the data of every user that no longer exists there. The walk includes the first-completion records, so it also reaches an account that only completed lesson units, and the records that earlier versions left behind for deleted accounts.

A user counts as deleted only if the auth microservice answers 404 with its UserNotFoundError ({"detail": "User not found"}). Any other answer, including a 404 for a route that does not exist, keeps the user's data and counts as an error. A lookup gives up after 10 seconds. Each failed lookup is logged at info level, and the run adds one warning with the number of failed lookups. If more users are missing, or more lookups fail, than both max_missing and max_missing_share of the checked users allow, a broken auth microservice is more likely than that many lost deletions: the run deletes nothing, logs an error with the counts and exits with a failure. After checking that these accounts really have been deleted and why the lookups failed, run the sweep once with a higher limit (see below for hosts with the NixOS module).

The settings live in the [deleted_user_sweep] section of config.toml and can be overridden with environment variables (__ separates the section from the property):

Property	Environment variable	Default	Description
batch_size	DELETED_USER_SWEEP__BATCH_SIZE	500	Number of user ids loaded from the database per batch.
rate_limit	DELETED_USER_SWEEP__RATE_LIMIT	10	Auth microservice requests per second; 0 means unlimited.
max_missing	DELETED_USER_SWEEP__MAX_MISSING	25	Number of missing users, and separately of failed lookups, a run always accepts.
max_missing_share	DELETED_USER_SWEEP__MAX_MISSING_SHARE	0.01	Share of the checked users that may be missing, and separately fail to be looked up, in a run, if that is more than max_missing.

The base url of the auth microservice is services.auth (SERVICES__AUTH).

In the NixOS module the sweep is the oneshot service academy-challenges-sweep-deleted-users with a timer, enabled through academy.backend.challenges.sweepDeletedUsers.enable (interval, default daily, randomizedDelay, default 5m, and timeout, default 2h, after which systemd stops the run and marks the unit as failed). The service provides the configuration file, the secrets and the database user, so on such a host the sweep runs through the service and not by calling the binary. To run it once by hand:

systemctl start academy-challenges-sweep-deleted-users

To run it once with a higher limit, put the override into a runtime drop-in, start the run without waiting for it, and remove the drop-in as soon as the run's process has started; the process keeps the environment it started with. The drop-in then exists only for a few seconds, also if the shell session ends during the run. A drop-in that stayed would raise the limit for every following night until the next reboot. While the unit is idle, ExecMainPID still names the process of the previous run, so the script waits for a new ExecMainStartTimestampMonotonic instead. The script accepts a unit in the state failed, which is what the aborted run that calls for a catch-up leaves behind; systemctl start needs no reset-failed before. If a step fails before the drop-in is removed, the trap removes it when the script ends. Run it as a script, for example through ssh <host> bash -s:

set -eu
u=academy-challenges-sweep-deleted-users
case "$(systemctl show -p ActiveState --value "$u")" in
  inactive|failed) ;;
  *) echo "$u is busy" >&2; exit 1 ;;
esac
before=$(systemctl show -p ExecMainStartTimestampMonotonic --value "$u")
trap 'systemctl revert "$u"' EXIT
printf '[Service]\nEnvironment=DELETED_USER_SWEEP__MAX_MISSING=200\n' \
  | systemctl edit --runtime --stdin --drop-in=catch-up "$u"
systemctl start --no-block "$u"
for _ in $(seq 300); do
  test "$(systemctl show -p ExecMainStartTimestampMonotonic --value "$u")" != "$before" && break
  sleep 0.2
done
systemctl revert "$u"
while test "$(systemctl show -p ActiveState --value "$u")" = activating; do sleep 5; done
systemctl show -p Result -p ExecMainStatus "$u"

License

Copyright (C) 2023–2026 Cedric Mössner

This project is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. See LICENSE for the full license text.