Skip to content

Feature/1035 lua delete - #1083

Open
nagisml wants to merge 22 commits into
OpenSAK-Org:betafrom
nagisml:feature/1035-lua-delete
Open

nagisml wants to merge 22 commits into
OpenSAK-Org:betafrom
nagisml:feature/1035-lua-delete

Conversation

@nagisml

@nagisml nagisml commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Lua macros can now delete caches and their rows, keeping the database consistent with the GUI delete.

Merge order: this branch is based on feature/lua-cache-write and must be merged after #1080. Until then the diff here also shows #1080's commits.

Closes #1035

Changes

New macro functions (behind the lua-macro-write flag; they need the same per-macro write permission as opensak.update()):

Function Deletes Returns
opensak.delete(code) a cache with its logs, waypoints, attributes, trackables and user note false if not in the database
opensak.delete(codes) the same for a list, e.g. opensak.delete(opensak.codes()) number deleted
opensak.delete_log/waypoint/attribute/trackable(id) one row; the cache's counts and log dates are recalculated false if no such row

opensak.sql_write() allows DELETE on the cache tables (caches, logs, waypoints, attributes, trackables, user_notes):

  • The schema's foreign keys have no ON DELETE CASCADE. A temporary BEFORE DELETE ON caches trigger deletes a cache's rows first, so there is no schema change, no migration and no FOREIGN KEY constraint failed error.
  • AFTER DELETE triggers on the child tables record the cache, so its counts and log dates are recalculated after the statement. Inside a transaction that happens at the commit or on opensak.resync().
  • INSERT OR REPLACE on a cache now replaces it and its rows. It used to be refused.

Consistency and safety

  • Deletes take part in opensak.transaction() and are undone with it, including nested transactions.
  • The GUI uses the existing caches_removed() host hook, which reloads the list and map and clears the detail panel when the macro ends.
  • At the end of a run the macro output reports what was deleted, e.g. opensak: 3 caches deleted from database 'X'. Deletes a transaction undid are not counted.

Refactoring: the main window's bulk delete and move_caches had the same delete code copied. Both now use db/cache_delete.delete_caches(), like the macro functions.

Decisions on the open questions in #1035

  1. Separate confirmation for cache deletes: no. The existing write permission (Deny / Until OpenSAK closes / Always) covers deletes.
  2. Bulk delete without SQL: yes. opensak.delete() accepts a list of codes.
  3. GC codes stored elsewhere: none. Only the six cache tables reference caches.
  4. Deleting the cache selected in the GUI: the selection is cleared, as after moving caches.

Correction to the issue: the touched-cache triggers did not record deletes before. The delete triggers only refused them. This PR adds that recording.

Docs

The API registry and the "Changing caches" and "Logs, waypoints, attributes and trackables" sections are updated. docs/macros/api.md and the Lua stub are regenerated by the Macro API docs workflow after the merge.

Tests

  • New tests/unit-tests/test_macro_delete_1035.py (24 tests): the cascade with no orphaned rows, list deletes, child-row deletes and recalculated counts, SQL deletes, INSERT OR REPLACE, transaction commit and rollback (nested too), the GUI notification, the end-of-run report, write permission, and the refusal paths.
  • Existing tests adjusted where they relied on DELETE being refused. The flag test now also covers the delete functions.
  • Full unit suite: 6005 passed, 108 skipped.

Inside opensak.transaction() the columns OpenSAK maintains (distance,
bearing, log/waypoint/trackable counts, log dates) are no longer
recalculated after every write. The touched caches are collected and
recalculated once, just before the commit. This covers sql_write() and
the ORM writes (update, insert, insert_/update_<child>).
opensak.resync() recalculates them on demand inside a transaction.

Codes of caches added by SQL INSERT are still normalised immediately,
so the cache can be found by its code inside the transaction. A
rolled-back transaction or savepoint leaves nothing pending, and a
failed recalculation undoes the whole transaction. Outside a
transaction, every write is still recalculated before its own commit.

2000 insert_log calls on one cache in a transaction: 4.5 s -> 1.8 s.
- New feature flag lua-macro-write (beta builds only): update, insert,
  insert_*/update_*, sql_write, transaction, in_transaction and resync
  are nil without it; set_corrected/clear_corrected then change the
  database without asking, as before. The Settings list of approved
  databases is hidden too.
- Docs steer authors to update/insert/insert_<x>/update_<x> first and
  present sql_write as the tool for bulk work; gated functions are
  marked beta.
- opensak.macro_db() checks write access to the macro's folder through
  the folder-permission model (OpenSAK-Org#978), prints the path when it creates
  the file, and a read-only macro database gives a clear error.
- Test fixture: reset_flags restores features.json before reloading,
  so patched flag values no longer leak into later tests.
opensak.macro_db() built the file name from the called macro's folder and
the caller's chunk name, so a macro started with opensak.run_macro()
opened <caller>.db3 in the callee's folder (OpenSAK-Org#1039).

- Name the file after _macro_path (switched by run_macro()); fall back to
  base_dir / chunk_name for a top-level run without a macro path.
- In a called macro, macro_db() detaches the caller's file and attaches
  the callee's own as `macro`; the caller's file is attached again when
  the called macro returns, also after an error (the callee's error wins
  over a failed re-attach).
- Until the callee calls macro_db(), `macro` still means the caller's
  file (DETACH is impossible inside opensak.transaction()).
- WritableDatabase.detach_macro_db(); macro_db() docs updated.
- Test: caller and helper in lib/, normal and failed return.
…#1034)

Before a macro's first write, OpenSAK now asks whether this macro may
change the OpenSAK databases (Deny / Until OpenSAK closes / Always).
"Always" is stored by macro file path (macros.write_always) and covers
all databases; macros still only write the active one. Editing the
macro keeps the approval.

- Unsaved macros get Deny / Until OpenSAK closes only.
- Macros started with run_macro() are asked about on their own.
- set_corrected/clear_corrected use the same approval.
- Settings → Folder permissions lists approved macros by name
  (path in tooltip), removable.
- MacroHost.approve_database_write(name, path) is now
  approve_macro_write(macro).
- New dialog texts in all languages; docs and tests updated.
…OpenSAK-Org#1035)

New macro functions (behind the lua-macro-write flag, write permission
as for opensak.update()):
- opensak.delete(code | codes): deletes caches with their logs,
  waypoints, attributes, trackables and user note. One code returns a
  boolean; a list returns the number deleted.
- opensak.delete_log/waypoint/attribute/trackable(id): deletes one row
  and recalculates the cache's counts and log dates.

opensak.sql_write() now allows DELETE on the cache tables. The schema's
foreign keys have no ON DELETE CASCADE, so a temporary BEFORE DELETE
trigger on caches deletes a cache's rows first. AFTER DELETE triggers on
the child tables record the cache for the derived-column refresh. Also
covers INSERT OR REPLACE, which now replaces a cache and its rows
instead of being refused.

Deletes take part in opensak.transaction() and are undone with it. The
GUI reloads through the existing caches_removed() host hook, which also
clears the detail panel. At the end of a run the macro output reports
the number of deleted caches per database, without deletes a
transaction undid.

The bulk delete of the main window and of move_caches now share
db/cache_delete.delete_caches().

Docs (API registry, Changing caches section) updated. The generated
api.md and Lua stub are refreshed by CI. Tests: test_macro_delete_1035.py,
plus existing tests adjusted for DELETE no longer being refused.
@AgreeDK

AgreeDK commented Oct 18, 2026

Copy link
Copy Markdown
Member

Thanks @nagisml — the delete functions themselves look good. Logs, waypoints, attributes, trackables and the user note are cleaned up, deletes roll back with opensak.transaction(), the end-of-run summary reports them, and sharing db/cache_delete.delete_caches() with the GUI bulk delete is a nice cleanup.

Requesting one change: INSERT OR REPLACE. Before this PR it was refused. Now it deletes the cache's logs, waypoints, attributes, trackables and personal note, then inserts a fresh row, and (per test_sql_insert_or_replace_replaces_the_cache_and_its_rows) it isn't reported as a delete. Many people use INSERT OR REPLACE as an "update or insert", GSAK users included. A macro meant to change one field would silently wipe a cache's data, and personal notes can't be recovered from a GPX.

Could we keep refusing REPLACE (including REPLACE INTO and ON CONFLICT REPLACE) with an error pointing to INSERT ... ON CONFLICT(gc_code) DO UPDATE SET ...? That's a real upsert, and it already goes through the existing INSERT/UPDATE authorizer.

Question: macros can delete (and update) locked caches. Locking means "protect from import", so this may well be intended — but I'd like it to be a deliberate decision and mentioned in the docs.

This PR is stacked on #1080, so I'll merge it after that one.

Root bypasses POSIX file permissions, so chmod(S_IREAD) can't make the
file read-only and the write-rejection assertions fail. Skip the test
when euid == 0 (reported in PR review).
REPLACE resolves a conflict by deleting the existing row, and the cache
delete trigger then removes the cache's logs, waypoints, attributes,
trackables and user note. sql_write now refuses INSERT OR REPLACE,
REPLACE INTO, UPDATE OR REPLACE and ON CONFLICT REPLACE before the
statement runs, and points to INSERT ... ON CONFLICT (gc_code) DO UPDATE.
@nagisml

nagisml commented Oct 18, 2026

Copy link
Copy Markdown
Contributor Author

@AgreeDK realigned branch on the lua write branch

  • INSERT OR REPLACE: Good catch, thanks. Changed as suggested: sql_write now refuses REPLACE before the statement runs. That covers INSERT OR REPLACE, REPLACE INTO and ON CONFLICT REPLACE, and also UPDATE OR REPLACE, which can delete a conflicting row the same way. The error points to INSERT ... ON CONFLICT (gc_code) DO UPDATE SET ....

The authorizer doesn't report the conflict resolution, so this is a keyword check that skips comments, strings and quoted names. The replace() function still works. The schema has no ON CONFLICT REPLACE constraints, so the keyword forms are the only way in.

The old test now expects all forms to be refused and checks that the cache, its logs and its note are untouched. A new test shows the ON CONFLICT ... DO UPDATE upsert keeps the child rows. The sql_write API docs mention the refusal and the alternative.

  • Lock: Yes, that's deliberate. The lock flag protects a cache from being overwritten by an import, and that's all it does. Macros are something the user runs on purpose, with write permission they granted, so they can update and delete locked caches the same way the GUI can. A full lock that also blocks macro writes and deletes could be added if users ask for it. It would have to be enforced in every write path (update(), the child writers, delete(), and the authorizer and triggers for sql_write), so I'd rather not add that complexity without real demand.

I'll add a note to the docs for delete(), update() and sql_write saying the lock only protects against imports and doesn't stop macro changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants