feat(evals): add SPACE text map sketching with official scoring by kellyang135 · Pull Request #4540 · dimensionalOS/dimos

kellyang135 · 2026-10-12T04:50:44Z

Why do we need this?

Map sketching tests how a model combines observations from a walk into an
overhead layout. SPACE's text presentation fits the existing eval framework
and provides objective grading without an image pipeline or simulator.

What is this feature?

Adds SPACE's MapSketchingBEVText task to EvalRunner with a fixed 20-question
subset. The prompt-only agent reuses dimOS's single-call and tracing path;
questions and answer order are unchanged.

The space extra installs ml-space-benchmark-ky135==0.1.1, an unofficial
package of unchanged upstream source. Setup verifies the installed scorer and
downloads pinned task data. SPACE's unmodified evaluator scores saved replies.
An exclude-newer-package exception permits this new release under the
repository's seven-day cutoff; the version remains pinned.
See task selection and setup.

How to manually run this

With Linux or macOS, Python 3.12, and an OpenAI API key configured:

uv sync --locked --no-default-groups --extra space
uv run --no-sync dimos evals setup space
uv run --no-sync dimos evals run dimos.evals.suites.space_map_sketching \
  --agent dimos.evals.agents.text_question \
  --set model=gpt-5.6-luna --set max_output_tokens=16384
uv run --no-sync dimos evals score space /path/to/run-directory

Setup prints the 20 selected cases. Use the directory printed by the run
command for scoring. The score command writes report.json in a new
space-score-* subdirectory with the official accuracy, per-case scores, and
invalid-response and failure counts. Scoring makes no provider calls.

Evaluation report

Configuration: gpt-5.6-luna alias, medium reasoning, a 16,384-token output cap,
original question only, no system prompt or tools, zero retries, and a 60-second
provider timeout.

The fixed subset contains two variants of each of ten layouts, balanced across
answer positions. Official accuracy was 20/20 (100%) in one recorded model
run, with no invalid replies, timeouts, or infrastructure failures. These related
cases do not establish performance across SPACE.

The output cap was increased after a lower cap produced truncated replies
without answers. Repeat runs at intermediate caps would test token efficiency;
evaluating the remaining task variants would extend coverage beyond this subset.

Which issue(s) does this PR close?

Addresses #3399.

Checklist

I have read and approved the CLA.

Reuse SingleCallAgent for original-question requests without observations or tools. Allow explicit request limits and close owned tracing clients. Cover both OpenAI request formats, failures, timeouts, and existing eval agents.

Acquire pinned SPACE inputs and run a fixed subset through EvalRunner. Score saved replies with the unmodified official evaluator. Include setup instructions and tests for acquisition, scoring, and failures.

greptile-apps · 2026-10-12T05:02:11Z

Confidence Score: 5/5

[Medium impact] No blocking or non-blocking defects were identified; the reviewed changes are safe to merge based on the examined code and successful execution checks.

T-Rex evidence

Evidence from the check

The executed command installs Python 3.12 into the local setup directory so the requested interpreter can be restored.

Command output from the check

Captured installation output shows Python 3.12.15 installed successfully, resolving the missing interpreter dependency.

Evidence from the check

The executed command reconnects the existing virtual environment to the recovered interpreter without changing product source.

Command output from the check

Captured output verifies that the required virtual environment interpreter runs Python 3.12.15 successfully.

Evidence from the check

The executed pytest command selects only original-prompt and limits, timeout, and server-failure tests from the requested file.

Command output from the check

Captured pytest output shows all four selected local HTTP provider tests passed, supporting the scoped head verdict.

Evidence from the check

The executed harness reuses the existing HTTP fixture with synthetic prompts and checks both SDK endpoints under success, timeout, and 503 conditions.

Evidence from the check

The executed command runs the uploaded harness with the requested virtual environment interpreter in either historical or head mode.

Command output from the check

Captured historical function execution records both endpoints and all three conditions, showing omitted limits, 503 retries, and unclosed clients before the change.

Command output from the check

Captured head execution records HTTP statuses, provider messages, trace retention, single attempts, and client closure, with all 110 assertions passing.

Command output from the check

Ran the captured uv installation command after discovering a dangling venv interpreter symlink; Python 3.12.15 was restored successfully, allowing the requested tests to execute.

Command output from the check

Executed the same two pytest commands in an isolated parent-revision checkout; both requested files were absent, so no before-change behavioral comparison was possible.

Evidence from the check

Executed this script in head and base modes to run only the requested checks and capture their real output; it contains the exact test commands without exporting benchmark content.

Command output from the check

Executed four official integration tests and 14 narrow report tests with canned replies and the pinned cache; all 18 tests passed and the scoped hypothesis was not reproduced.

Summary

Adds SPACE's text map-sketching task to EvalRunner, with pinned setup inputs and official scoring of saved replies.

dimOS now runs SPACE’s fixed set of text map-sketching questions.
Saved SPACE replies now produce an official score report.
Text questions now run through a prompt-only, single-call agent.

No actionable issues were found. Four local HTTP provider tests passed, covering original prompts, output limits, timeout handling, and server failures. Additional checks passed across six scenarios on both provider endpoints, including single-request behavior and client cleanup. All four official-evaluator integration tests and 14 report tests passed, covering saved-reply scoring and handling of missing, failed, or mismatched results. These checks used canned replies and local HTTP responses; no live model provider was used.

The reviewed changes are safe to merge based on the examined code and these checks.

_{Reviews (1) · Last reviewed commit: "feat(evals): integrate SPACE text map sk..." · Reviewed by Greptile}

codecov · 2026-10-14T20:42:41Z

Codecov Report

❌ Patch coverage is 92.02454% with 130 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/evals/suites/lib/space/test_official.py	31.86%	62 Missing ⚠️
dimos/evals/suites/lib/space/report.py	88.69%	9 Missing and 10 partials ⚠️
dimos/evals/suites/lib/space/upstream.py	84.26%	7 Missing and 7 partials ⚠️
dimos/evals/suites/lib/space/worker.py	78.57%	11 Missing and 1 partial ⚠️
dimos/evals/suites/lib/space/suite.py	86.04%	5 Missing and 1 partial ⚠️
dimos/agents/mcp/mcp_client.py	66.66%	2 Missing and 2 partials ⚠️
dimos/evals/agents/lib/single_call.py	91.17%	1 Missing and 2 partials ⚠️
dimos/evals/suites/lib/space/test_commands.py	95.00%	1 Missing and 2 partials ⚠️
dimos/evals/suites/lib/space/test_data.py	98.32%	2 Missing and 1 partial ⚠️
dimos/evals/suites/lib/space/commands.py	93.54%	1 Missing and 1 partial ⚠️
... and 1 more

@@            Coverage Diff             @@
##             main    #4540      +/-   ##
==========================================
+ Coverage   80.44%   80.56%   +0.12%     
==========================================
  Files        1655     1675      +20     
  Lines      155298   156914    +1616     
  Branches    13120    13245     +125     
==========================================
+ Hits       124922   126424    +1502     
- Misses      26830    26921      +91     
- Partials     3546     3569      +23

Components	Coverage Δ
Tests	`95.28% <93.01%> (-0.04%)`	⬇️

Flag	Coverage Δ
OS-macos-latest	`74.97% <92.02%> (+0.18%)`	⬆️
OS-ubuntu-24.04-arm	`74.73% <92.02%> (+0.18%)`	⬆️
OS-ubuntu-latest	`75.09% <92.02%> (+0.18%)`	⬆️
Py-3.10	`75.09% <92.02%> (+0.18%)`	⬆️
Py-3.11	`75.08% <92.02%> (+0.17%)`	⬆️
Py-3.12	`75.09% <92.02%> (+0.17%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/evals/agents/test_text_question.py	`100.00% <100.00%> (ø)`
dimos/evals/agents/text_question.py	`100.00% <100.00%> (ø)`
dimos/evals/environments/prompt.py	`100.00% <100.00%> (ø)`
dimos/evals/suites/lib/space/constants.py	`100.00% <100.00%> (ø)`
dimos/evals/suites/lib/space/data.py	`100.00% <100.00%> (ø)`
dimos/evals/suites/lib/space/process.py	`100.00% <100.00%> (ø)`
dimos/evals/suites/lib/space/test_process.py	`100.00% <100.00%> (ø)`
dimos/evals/suites/lib/space/test_report.py	`100.00% <100.00%> (ø)`
dimos/evals/suites/lib/space/test_suite.py	`100.00% <100.00%> (ø)`
dimos/evals/suites/lib/space/test_worker.py	`100.00% <100.00%> (ø)`
... and 13 more

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

paul-nechifor · 2026-10-14T22:57:31Z

+    raise SystemExit(128 + signum)
+
+
+def main(argv: Sequence[str] | None = None) -> int:


I don't think we should have random executable files in the repo.

It's best to add it as a subcommand in dimos. Something like dimos evals setup space (just an idea, not sure about the exact command).

paul-nechifor · 2026-10-14T23:05:17Z

+    return reader.bytes_read
+
+
+def setup(paths: SpacePaths) -> dict[str, Any]:


I don't think we should be cloning repos. A better alternative is for us to package their repo and publish it to PyPI.

It looks like they don't even specify versions for their dependencies: https://github.com/apple-aiml-research/ml-space-benchmark/blob/main/requirements.txt . This is bound to give us headaches in the future.

ok, I understand! I’ve prepared and tested the SPACE package and updated dimOS to depend on it, following the edgetam-dimos approach I saw in a previous PR.
Where would you like me to submit the packaging changes for review and a PyPI release?

Maybe just link it here, I'll have a quick look.

But yes, please release to PyPI. (For the package name, you usually want to prefix/suffix it with your name/identifier so an official version of SPACE can be released by them if they choose to.)

So here is the published ml-space-benchmark-ky135 0.1.1; package source and docs. It preserves Apple’s SPACE source and adds dependency version bounds.
This PR now installs it through the space extra instead of cloning the repository.

paul-nechifor · 2026-10-14T23:07:25Z

 ]

+# Benchmark dependencies for SPACE runs and self-hosted integration tests.
+space = [


I don't think we should list their dependencies. It's best to package their repo and publish it to PyPI and just depend on the package.

paul-nechifor · 2026-10-15T00:25:48Z

+    state = ProviderState()
+    state.release.set()
+
+    class Handler(BaseHTTPRequestHandler):


You can use httpx.MockTransport like test_llm_trace.py instead of creating an HTTP server

hvent90

Really good. Left a few comments. Thanks for the great PR, Kelly!

hvent90 · 2026-10-22T21:15:19Z

+    from dimos.evals.agents.base import Agent
+
+
+class Prompt(Environment):


hvent90 · 2026-10-22T21:21:19Z

+    max_output_tokens: int | None = Field(default=2048, gt=0)
+
+
+class TextQuestion(SingleCallAgent):


Let's have SingleCallAgent fulfill this role - just taking in the prompt with no special processing and sending it to the LLM provider. I don't think we need to throw an error if streams or mcp_urp exist, it will just ignore it.

And then dimos/evals/agents/question_answer.py extends SingleCallAgent and continues to do the processing of streams and adding that to the request sent to the LLM

hvent90 · 2026-10-22T21:23:19Z

+
+```bash skip
+uv sync --locked --no-default-groups --extra space
+uv run --no-sync dimos evals setup space


Lets co-locate functionality with the domain by doing [noun] [verb] sequencing, so it would be:

dimos evals space setup dimos evals space score

kellyang135 added 2 commits October 11, 2026 19:07

feat(evals): support prompt-only evaluations

b1d790f

Reuse SingleCallAgent for original-question requests without observations or tools. Allow explicit request limits and close owned tracing clients. Cover both OpenAI request formats, failures, timeouts, and existing eval agents.

feat(evals): integrate SPACE text map sketching

8152b48

Acquire pinned SPACE inputs and run a fixed subset through EvalRunner. Score saved replies with the unmodified official evaluator. Include setup instructions and tests for acquisition, scoring, and failures.

kellyang135 requested review from Dreamsorcerer, TomCC7, aclauer, leshy, mustafab0, paul-nechifor and spomichter as code owners October 12, 2026 04:50

github-actions Bot added the first-time-contributor PR opened by an author who had not previously committed to this repository label Oct 12, 2026

paul-nechifor reviewed Oct 14, 2026

View reviewed changes

paul-nechifor reviewed Oct 15, 2026

View reviewed changes

kellyang135 added 8 commits October 14, 2026 20:14

Merge origin/main into contributor/feat/space-map-sketching

1579f94

fix(evals): integrate SPACE setup and scoring into the CLI

3554a03

test(evals): use MockTransport for provider responses

52d8060

refactor(evals): use packaged SPACE scorer

7a63feb

docs(evals): trim SPACE test description

26cfd90

build(evals): update SPACE package to 0.1.1

c84c041

refactor(evals): use installed SPACE package metadata

6afc7df

docs(evals): tighten SPACE comments and usage notes

ed1d528

hvent90 requested changes Oct 22, 2026

View reviewed changes

kellyang135 added 3 commits October 22, 2026 17:28

chore(evals): merge main

1ca3d3a

refactor(evals): make SingleCallAgent directly usable

3d61e29

refactor(evals): group SPACE commands by benchmark

afc3557

		raise SystemExit(128 + signum)


		def main(argv: Sequence[str] \| None = None) -> int:

		return reader.bytes_read


		def setup(paths: SpacePaths) -> dict[str, Any]:

		from dimos.evals.agents.base import Agent


		class Prompt(Environment):

		max_output_tokens: int \| None = Field(default=2048, gt=0)


		class TextQuestion(SingleCallAgent):

Conversation

kellyang135 commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why do we need this?

What is this feature?

How to manually run this

Evaluation report

Which issue(s) does this PR close?

Checklist

Uh oh!

greptile-apps Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

T-Rex evidence

Summary

Uh oh!

codecov Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hvent90 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kellyang135 commented Oct 12, 2026 •

edited

Loading

greptile-apps Bot commented Oct 12, 2026 •

edited

Loading

codecov Bot commented Oct 14, 2026 •

edited

Loading