Stream GPX/LOC/GGZ exports to avoid running out of memory, fixes #1075

[nagisml]

Fixes #1075

Exporting a large database (100,000+ caches) to GPX, LOC or GGZ ran out of memory. Every cache was reloaded with its logs, waypoints and descriptions at once, and then the whole document was built in memory before it was written.

What changes

• Chunked reload: the new iter_caches_full() reloads caches one chunk at a time. reload_caches_full() is now just list(iter_caches_full(...)).
• Streaming writers: write_gpx / write_loc / write_ggz replace generate_* and write each cache as it is reloaded. GGZ spools the GPX and the index into temporary files and then builds the zip.
• Safe replace: file export writes to <name>.part and renames it over the target only when it has finished. If the export fails, the existing file is kept and no truncated file is left behind.
• Other callers: device export (the GPS dialog) and POI export use the same streaming path.

Known limitations / follow-ups

• Garmin in mass-storage mode: for a short time the old and the new file both exist on the device, where before the file was overwritten in place. On a nearly full device, an export that used to work could now fail. This is an edge case and doesn't block the PR.
• KML: still loads all caches at once. Left for a later PR.

[AgreeDK]

Thanks @nagisml — nice fix. Streaming the export chunk by chunk, and writing to a .part file that only replaces the target once complete, is exactly right for #1075.

Since GSAK byte-for-byte compatibility matters a lot for GGZ/GPX, I checked it explicitly. I generated GPX, LOC and GGZ with the old and new code from the same test data (æøå, &, <>, child waypoints, logs, a cache without coordinates):

• GPX and LOC are byte-for-byte identical.
• The GGZ index differs only in the CRC, because the GPX contains the export timestamp. In both versions the CRC matches the embedded GPX.

The export, GPS dialog, POI and DB tests (380) all pass.

Minor:

• Could the title and commit message say what the change does, and include fixes #1075? E.g. "Stream GPX/LOC/GGZ exports to avoid running out of memory (Memory error when exporting to GPX or LOC #1075)".
• On a Garmin in mass-storage mode, the old and new file now exist side by side for a moment, where the file used to be overwritten in place. On a nearly full device an export that worked before could now fail. It's an edge case and not blocking; I'm only noting it.
• KML still loads everything at once. Fine to leave for later.

Approving.

[nagisml]

@AgreeDK PR & commits fixed