Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .sampo/changesets/virtuous-shaman-kalma.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
pypi/posthog: minor
---

Update and delete offline experiment items, reported scores, and payloads from sync and async clients using the current revision.
44 changes: 42 additions & 2 deletions examples/example-offline-evaluations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ Offline evaluations must be enabled for your PostHog project. Use the project's
| Create/update/archive/restore datasets and items | Personal API key with `dataset:write` |
| Read experiment tags | Personal API key with `evaluation:read` |
| Replace experiment tags | Personal API key with `evaluation:write` |
| Read an item or result revision | Personal API key with `evaluation:read`. Result reads also require `llm_analytics:read` and access to the scorer. |
| Update an item or update/delete its payload | Personal API key with `evaluation:write` and evaluation editor access |
| Delete an item, change/delete a result or its payload | Personal API key with `evaluation:write`, evaluation editor access, `llm_analytics:read`, and access to affected scorers and composite inputs |

A personal key can cover both scorer management and uploads. Project secret keys can upload using a known scorer-version UUID without scorer read access. All operations use the explicit project/environment; scorer names are not unique.

Expand Down Expand Up @@ -80,7 +83,7 @@ with OfflineEvaluations(

Use one `EvaluationItem` for the application's execution and reuse it across scorer results. `upload_results([EvaluationResult(...), ...])` uploads a finite sequence, eagerly and sequentially. It splits at 1,000 items/results or 5 MiB of UTF-8 JSON per request, whichever comes first. Item payloads are limited to 1 MiB; result payloads to 256 KiB. Payload JSON supports at most 32 nested containers. The SDK raises for invalid values and oversized payloads without truncating them. Memory use scales with the sequence you supply, so submit caller-sized batches for large datasets.

Set `model_version` on each item to compare executions from different models in one experiment. It identifies the application's model and overrides the default supplied to `create_experiment(model_version=...)`. Omit it or pass `None` to inherit that default on the server; without either value, the model remains unknown. Explicit model identifiers must be nonempty strings of at most 255 characters and are preserved without trimming. The declared model is immutable item content: changing, adding, or removing it after acceptance causes a `content_conflict`, even if the effective model is unchanged. Create a new item for a different model execution, and preserve the original declaration for retries.
Set `model_version` on each item to compare executions from different models in one experiment. It identifies the application's model and overrides the default supplied to `create_experiment(model_version=...)`. Omit it or pass `None` to inherit that default on the server; without either value, the model remains unknown. Explicit model identifiers must be nonempty strings of at most 255 characters and are preserved without trimming. Re-uploading a retained item with a different declaration raises `content_conflict`. Use `update_item` to correct a recorded model, and create a new item for a different execution. Keep the original declaration for upload retries.

`status="ok"` means the evaluator ran successfully. Its boolean, number, or category-list `value` is uploaded with its original polarity. Numeric scores use finite double-precision (binary64) values; integer inputs are rejected if conversion would change their value. Use `error`, `skipped`, or `not_applicable` without a value for other outcomes. `error_code` and a non-null `error_message` are valid only with `status="error"`.

Expand Down Expand Up @@ -152,7 +155,7 @@ The SDK keeps all supplied results for a composite-bearing item together without

HTTP success acknowledges persistence, not successful calculation. Input errors become `composite_input_error`; formula errors or invalid output values become `composite_formula_error`. Input statuses propagate in the order `error`, `skipped`, `not_applicable`. Check each receipt's `status` and `error_code`; no result-read permission is needed. For ordinary results on an older backend, `status=None` means the backend omitted the outcome. Composite acknowledgments require an outcome.

All accepted composite outcomes count toward `expected_result_count`, including errors and skips. Derived results add no uploaded-result billing usage. Omitting a composite leaves it unrequested. Exact retries return the stored outcome, including after completion; formulas are not recalculated on retry or read. Persist declarations with `to_dict` and restore them with `from_dict` for exact recovery.
All accepted composite outcomes count toward `expected_result_count`, including errors and skips. Derived results add no uploaded-result billing usage. Omitting a composite leaves it unrequested. An exact upload retry for a retained result returns its original outcome, even after an edit or experiment completion. The server does not recalculate formulas on retry or read. Persist declarations with `to_dict` and restore them with `from_dict` for exact recovery.

Formulas support 1–16 reported inputs and at most 1,024 characters; aliases start with a lowercase letter and contain lowercase letters, digits, or underscores, up to 32 characters. Nested composites are unsupported. Use `number(flag)` for explicit boolean conversion and `"good" in categories` for categorical membership. Boolean inputs retain their raw meaning regardless of passing rules. Numeric composites support bounds and passing thresholds, but no non-null `step`. Formula syntax, input access, and output types are validated by the server; runtime values and bounds are checked during ingestion.

Expand Down Expand Up @@ -425,6 +428,43 @@ Reading and writing experiment tags require the personal-key scopes listed above

The async client exposes the same API: await scorer methods and `experiment.get_tags()` / `experiment.set_tags(...)`.

## Correcting or deleting accepted data

Use the item and result IDs from the upload receipt. You can make corrections while an experiment is uploading or after it completes or fails. With `AsyncOfflineEvaluations`, await each read, update, and delete.

```python
experiment = evaluations.resume_experiment(experiment_id)
item_id = receipt.items[0].id
result_id = receipt.results[0].id

revision = experiment.get_result_revision(result_id)
revision = experiment.update_result(result_id, revision=revision, status="ok", value=False)
experiment.update_result_payload(result_id, revision=revision, payload={"reasoning": "Reviewed by hand"})

revision = experiment.get_item_revision(item_id)
revision = experiment.update_item(item_id, revision=revision, model_version="v2")
experiment.update_item_payload(item_id, revision=revision, payload={"expected_output": None})
```

Each update returns the server's current revision, including when a score already has the requested value. A payload shares its owner's revision. Changes to a result or its payload can also advance the item revision. A stale write raises `EvaluationAPIError` with HTTP 409 and code `revision_conflict`. After a conflict or lost response, inspect the current data before deciding whether to write again. The SDK sends each mutation once.

If an update acknowledgment returns a revision below the one you submitted, the SDK raises `EvaluationAPIError` with code `invalid_response` and `persistence="unknown"`.

Payload updates replace only the supplied top-level fields. Use `None` to store JSON null. Edits keep the original 30-day retention deadline. Editing an item's input, output, or expected output can mark existing scores stale. To correct a composite, edit its reported inputs and the server recomputes dependent scores.

Passing `payload={}` stores an empty first payload or preserves the fields of an existing payload. Either write advances the owner's revision. A deleted or expired payload raises HTTP 409 `payload_unavailable` even for `{}`. When you change a result's status from `error` to another status, the server removes any stored `error_message` from its payload.

| Method | What it permanently removes |
| --- | --- |
| `delete_item(item_id, revision=...)` | The item, its results, and their payloads |
| `delete_result(result_id, revision=...)` | The result, its payload, and dependent composites |
| `delete_item_payload(item_id, revision=...)` | The stored item payload |
| `delete_result_payload(result_id, revision=...)` | The stored result payload |

Deletes return `None` on acknowledgment. Deleting a payload advances its owner's revision. Deleting a result or its payload also advances the parent item's revision. Before editing an affected record that still exists, read its new revision with `get_item_revision` or `get_result_revision`.

A deleted or expired payload cannot be restored on that record. Re-uploading a deleted item or result can recreate it while the experiment is open. A recreated reported result counts toward usage again.

## Receipts, errors, and recovery

Upload receipts contain persisted item/result UUIDs, original acceptance times, and `created` flags. `created=False` acknowledges an exact duplicate. A bulk upload returns `chunks`; each chunk commits independently. A malformed or incomplete acknowledgment raises rather than reporting success.
Expand Down
3 changes: 3 additions & 0 deletions posthog/ai/evaluations/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
OfflineEvaluations,
)
from ._errors import EvaluationAPIError
from ._mutations import ItemUpdate, ResultUpdate
from ._scorer_types import (
BooleanScorerConfig,
CategoricalPassingRule,
Expand Down Expand Up @@ -93,13 +94,15 @@
"ExperimentOptions",
"ExperimentReceipt",
"ItemReceipt",
"ItemUpdate",
"JSONValue",
"NumericPassingRule",
"NumericScorerConfig",
"OfflineEvaluations",
"ResultOptions",
"ResultReceipt",
"ResultStatus",
"ResultUpdate",
"RunSource",
"ScoreValue",
"Scorer",
Expand Down
170 changes: 170 additions & 0 deletions posthog/ai/evaluations/_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,28 @@
from ._async_datasets import AsyncDatasets
from ._datasets import Datasets
from ._errors import EvaluationAPIError
from ._mutations import (
ItemUpdate,
ResultUpdate,
_item_changes,
_item_path,
_item_payload_changes,
_patch_body,
_response_revision,
_result_changes,
_result_path,
_result_payload_changes,
)
from ._scorers import AsyncScorers, Scorers
from ._serialization import (
JSONValue,
_MAX_ITEMS,
_MAX_REQUEST_BYTES,
_MAX_RESULTS,
_count,
_encode,
_identifier,
_integer,
_timestamp,
_uuid,
)
Expand Down Expand Up @@ -282,6 +296,81 @@ def set_tags(self, tags: Sequence[str]) -> tuple[str, ...]:
)
return _experiment_tags(data, self.id)

def get_item_revision(self, item_id: str | UUID) -> int:
"""Read the current revision shared by an item and its payload."""
path = self._path + _item_path(item_id)
with self._lock:
data = self._transport.request("GET", path)
return _response_revision(data, identity=item_id)

def get_result_revision(self, result_id: str | UUID) -> int:
"""Read the current result revision from its payload endpoint."""
path = self._path + _result_path(result_id) + "payload/"
with self._lock:
data = self._transport.request("GET", path)
return _response_revision(data, identity=result_id)

def _patch(self, path: str, revision: int, changes: dict[str, Any]) -> int:
body = _patch_body(revision, changes)
with self._lock:
data = self._transport.request(
"PATCH", self._path + path, body=body, retry_safe=False
)
return _response_revision(data, minimum=revision)

def _delete(self, path: str, revision: int) -> None:
params: dict[str, str | int] = {"revision": _integer(revision, minimum=0)}
with self._lock:
self._transport.request(
"DELETE", self._path + path, params=params, retry_safe=False
)

def update_item(
self, item_id: str | UUID, *, revision: int, **fields: Unpack[ItemUpdate]
) -> int:
"""Update item identifiers at the observed revision and return the current revision."""
return self._patch(_item_path(item_id), revision, _item_changes(fields))

def update_result(
self, result_id: str | UUID, *, revision: int, **fields: Unpack[ResultUpdate]
) -> int:
"""Correct a reported score and return its revision. Dependent composites are recalculated."""
return self._patch(_result_path(result_id), revision, _result_changes(fields))

def update_item_payload(
self, item_id: str | UUID, *, revision: int, payload: dict[str, JSONValue]
) -> int:
"""Patch input, output, expected_output, or metadata and return the item revision."""
return self._patch(
_item_path(item_id) + "payload/", revision, _item_payload_changes(payload)
)

def update_result_payload(
self, result_id: str | UUID, *, revision: int, payload: dict[str, JSONValue]
) -> int:
"""Patch reasoning, error_message, or metadata and return the result revision."""
return self._patch(
_result_path(result_id) + "payload/",
revision,
_result_payload_changes(payload),
)

def delete_item(self, item_id: str | UUID, *, revision: int) -> None:
"""Permanently delete an item and all its results and payloads."""
self._delete(_item_path(item_id), revision)

def delete_result(self, result_id: str | UUID, *, revision: int) -> None:
"""Permanently delete a result, its payload, and dependent composites."""
self._delete(_result_path(result_id), revision)

def delete_item_payload(self, item_id: str | UUID, *, revision: int) -> None:
"""Permanently remove the stored payload while retaining the item and scores."""
self._delete(_item_path(item_id) + "payload/", revision)

def delete_result_payload(self, result_id: str | UUID, *, revision: int) -> None:
"""Permanently remove the stored payload while retaining its result."""
self._delete(_result_path(result_id) + "payload/", revision)

def upload_result(
self,
*,
Expand Down Expand Up @@ -412,6 +501,85 @@ async def set_tags(self, tags: Sequence[str]) -> tuple[str, ...]:
)
return _experiment_tags(data, self.id)

async def get_item_revision(self, item_id: str | UUID) -> int:
"""Read the current revision shared by an item and its payload."""
path = self._path + _item_path(item_id)
async with self._lock:
data = await self._transport.request("GET", path)
return _response_revision(data, identity=item_id)

async def get_result_revision(self, result_id: str | UUID) -> int:
"""Read the current result revision from its payload endpoint."""
path = self._path + _result_path(result_id) + "payload/"
async with self._lock:
data = await self._transport.request("GET", path)
return _response_revision(data, identity=result_id)

async def _patch(self, path: str, revision: int, changes: dict[str, Any]) -> int:
body = _patch_body(revision, changes)
async with self._lock:
data = await self._transport.request(
"PATCH", self._path + path, body=body, retry_safe=False
)
return _response_revision(data, minimum=revision)

async def _delete(self, path: str, revision: int) -> None:
params: dict[str, str | int] = {"revision": _integer(revision, minimum=0)}
async with self._lock:
await self._transport.request(
"DELETE", self._path + path, params=params, retry_safe=False
)

async def update_item(
self, item_id: str | UUID, *, revision: int, **fields: Unpack[ItemUpdate]
) -> int:
"""Update item identifiers at the observed revision and return the current revision."""
return await self._patch(_item_path(item_id), revision, _item_changes(fields))

async def update_result(
self, result_id: str | UUID, *, revision: int, **fields: Unpack[ResultUpdate]
) -> int:
"""Correct a reported score and return its revision. Dependent composites are recalculated."""
return await self._patch(
_result_path(result_id), revision, _result_changes(fields)
)

async def update_item_payload(
self, item_id: str | UUID, *, revision: int, payload: dict[str, JSONValue]
) -> int:
"""Patch input, output, expected_output, or metadata and return the item revision."""
return await self._patch(
_item_path(item_id) + "payload/", revision, _item_payload_changes(payload)
)

async def update_result_payload(
self, result_id: str | UUID, *, revision: int, payload: dict[str, JSONValue]
) -> int:
"""Patch reasoning, error_message, or metadata and return the result revision."""
return await self._patch(
_result_path(result_id) + "payload/",
revision,
_result_payload_changes(payload),
)

async def delete_item(self, item_id: str | UUID, *, revision: int) -> None:
"""Permanently delete an item and all its results and payloads."""
await self._delete(_item_path(item_id), revision)

async def delete_result(self, result_id: str | UUID, *, revision: int) -> None:
"""Permanently delete a result, its payload, and dependent composites."""
await self._delete(_result_path(result_id), revision)

async def delete_item_payload(self, item_id: str | UUID, *, revision: int) -> None:
"""Permanently remove the stored payload while retaining the item and scores."""
await self._delete(_item_path(item_id) + "payload/", revision)

async def delete_result_payload(
self, result_id: str | UUID, *, revision: int
) -> None:
"""Permanently remove the stored payload while retaining its result."""
await self._delete(_result_path(result_id) + "payload/", revision)

async def upload_result(
self,
*,
Expand Down Expand Up @@ -480,6 +648,8 @@ class OfflineEvaluations:
``secret_key`` accepts personal or project secret keys with
offline_evaluation_ingestion:write. Scorer reads/writes require a personal
key with llm_analytics:read/write; datasets require dataset:read/write.
Editing accepted data requires an evaluation editor's personal key with
evaluation:write. Result operations and item deletion also need scorer access.
Each operation requires its corresponding product feature to be enabled.
``host`` defaults to the US app API; known ingestion hosts are mapped to
their app hosts. Custom hosts are preserved. Requests use a 15 second
Expand Down
Loading