Repository navigation
fix(upload): keep an upload running in a foreground service and retry it by the kind of its error - #6828
fix(upload): keep an upload running in a foreground service and retry it by the kind of its error#6828ToteMeiSter wants to merge 22 commits into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Priority: ➖ Normal Severity of issue fixed: Medium Merge Risk: 🔵 Low · up to
|
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Organization UI
- Review profile: CHILL
- Plan: Advanced
- Run ID:
2b9b3b8e-5978-45b2-8d16-2f4f8d2a0b96
📒 Files selected for processing (20)
app/src/main/AndroidManifest.xmlapp/src/main/java/com/nextcloud/talk/chat/viewmodels/ChatViewModel.ktapp/src/main/java/com/nextcloud/talk/jobs/UploadAndShareFilesWorker.ktapp/src/main/java/com/nextcloud/talk/receivers/UploadCancelReceiver.ktapp/src/main/java/com/nextcloud/talk/upload/UploadNotification.ktapp/src/main/java/com/nextcloud/talk/upload/UploadProgressThrottle.ktapp/src/main/java/com/nextcloud/talk/upload/UploadRetryPolicy.ktapp/src/main/java/com/nextcloud/talk/upload/UploadWorkspace.ktapp/src/main/java/com/nextcloud/talk/upload/chunked/ChunkedFileUploader.ktapp/src/main/java/com/nextcloud/talk/upload/normal/FileUploader.ktapp/src/main/java/com/nextcloud/talk/utils/FileUtils.ktapp/src/main/java/com/nextcloud/talk/utils/ImageCompressor.ktapp/src/main/java/com/nextcloud/talk/utils/VideoCompressor.ktapp/src/test/java/com/nextcloud/talk/upload/UploadProgressThrottleTest.ktapp/src/test/java/com/nextcloud/talk/upload/UploadRetryPolicyTest.ktapp/src/test/java/com/nextcloud/talk/upload/UploadWorkspaceTest.ktapp/src/test/java/com/nextcloud/talk/upload/chunked/ChunkedFileUploaderAbortTest.ktapp/src/test/java/com/nextcloud/talk/upload/chunked/ChunkedFileUploaderResumeTest.ktapp/src/test/java/com/nextcloud/talk/upload/normal/FileUploaderTest.ktapp/src/test/java/com/nextcloud/talk/utils/FileUtilsContentResolverTest.kt
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 1 remain after this review.
|
@coderabbitai review |
|
|
@coderabbitai, on the two retained concerns of the summary:
Prepared with an AI assistant and reviewed by me. |
|
Tip For best results, initiate chat on the files or code changes.
Would you like me to open a follow-up issue for that false-failure case? ✏️ Learnings added
You are interacting with an AI system. |
|
This is a bug fix and all CodeRabbit review threads are resolved. Once it is approved and merged, would it be possible to backport it to Suggested command for a maintainer after merge: This comment was drafted with AI assistance (Claude Code). |
469e2bb to
4375439
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Organization UI
- Review profile: CHILL
- Plan: Advanced
- Run ID:
f22b0f6b-ba83-40a4-abd6-f6347d007754
📒 Files selected for processing (4)
app/src/main/java/com/nextcloud/talk/chat/viewmodels/ChatViewModel.ktapp/src/main/java/com/nextcloud/talk/upload/chunked/ChunkedFileUploader.ktapp/src/main/java/com/nextcloud/talk/utils/FileUtils.ktapp/src/test/java/com/nextcloud/talk/upload/chunked/ChunkedFileUploaderTest.kt
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review.
4375439 to
67d86dc
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Organization UI
- Review profile: CHILL
- Plan: Advanced
- Run ID:
ebeaba79-89fe-4361-9f17-79e3687bdf51
📒 Files selected for processing (6)
app/src/main/java/com/nextcloud/talk/jobs/UploadAndShareFilesWorker.ktapp/src/main/java/com/nextcloud/talk/upload/chunked/ChunkedFileUploader.ktapp/src/main/java/com/nextcloud/talk/utils/FileUtils.ktapp/src/test/java/com/nextcloud/talk/upload/chunked/ChunkedFileUploaderAbortTest.ktapp/src/test/java/com/nextcloud/talk/upload/chunked/ChunkedFileUploaderResumeTest.ktapp/src/test/java/com/nextcloud/talk/utils/FileUtilsContentResolverTest.kt
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.
67d86dc to
93d293a
Compare
rapterjet2004
left a comment
There was a problem hiding this comment.
Thanks, this is careful work: the own Dispatcher for stop(), ending failed works as success so the APPEND_OR_REPLACE queue isn't torn down, the cancel flag instead of cancelWorkById, and the stage markers in the workspace all look right. (Reviewed on top of #6827, i.e. only the commits of this PR.)
Change requested: a ceiling for the unlimited retries. The NETWORK class covers every 5xx except 507, 408, 429 and any IOException other than FileNotFoundException. A server that keeps answering 500 (broken app, misconfigured proxy), or a local read error on the prepared file, makes the upload retry forever: the placeholder stays "sending", a foreground notification shows on every run, and the prepared copy (possibly GBs) stays in noBackupFilesDir, until the user finds and cancels it. Waiting for the network without a limit is fine, but please give the "server temporarily unavailable" cases (5xx/408/429) a generous ceiling, e.g. a high count or a maximum total age since the first run (stored in the workspace, e.g. 24 h), after which the upload fails like a refusal.
Other points (non-blocking):
- Interaction with #6858: #6858 marks the placeholder failed when the work reaches
WorkInfo.State.FAILED. With this PR a failed upload ends asSUCCEEDED, so that branch never fires, and the case #6858 handles (the worker fails before the placeholder exists) would leave the placeholderPENDING. Whichever lands second needs to adapt, e.g. by reporting the failure in the work's output data. - Notification for every upload:
startForeground()runs before the preparation for every upload, so even a small photo flashes an "Uploading…" notification. Starting the foreground service only for chunked uploads (> 1 MiB), or after a short delay, would avoid that. - Play Console:
FOREGROUND_SERVICE_DATA_SYNCwith targetSdk 34+ needs a foreground service declaration in the Play Console. Just flagging it so it's handled before release. - Leftover draft file: a share that finally fails leaves the uploaded file in the draft folder:
failUpload→abortRemote()removes the parts butdeleteUploadedFileonly runs for a cancel.
… has A chunked upload restarted from zero after every interruption, although the parts it had sent were still on the server: - The PROPFIND on the upload folder named no properties. isFile is set only from a resourcetype property, so it stayed false for every member and the list of uploaded parts was always empty. The request now asks for resourcetype, getcontentlength and displayname. - Parts are accepted only when their name is well formed and their size matches the name. A wrong or foreign entry is uploaded again or ignored. - Only the missing parts are sent. - The assembling MOVE sends OC-Total-Length, so the server can check that the parts add up to the file. - A MOVE the server rejects with 400 removes the parts once and sends the file again. The caller supplies the "already restarted" mark. - stop() interrupts the upload between two parts and keeps the parts on the server. Only abortUpload() removes them. - upload() no longer swallows exceptions. A failed request is thrown (an HttpException for an HTTP error, an IOException for a network error) and the parts stay on the server. A lost network while listing the parts is thrown too instead of being taken for "no parts", which would send the whole file again. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
The folder of the parts on the server is keyed by the name, the modification time and the length of the file. UploadAndShareFilesWorker copied the content uri to the cache and compressed it again on every run. Each run produced a file with a new modification time, so a retry used another folder and sent the whole file again. The worker now prepares the file once into a directory of its own work (cacheDir/uploads/<work id>) and uploads the same file on every run: - UploadWorkspace keeps the prepared file and the "parts were removed once" mark, and holds a lock, because WorkManager does not interrupt the thread of a stopped worker and an old run can still be busy. - A workspace stays after a stop or before a retry. It is removed when the work ends, and the workspaces of finished or unknown works are removed before the next upload. - A run that is stopped while the file is prepared does not store the half-finished file. - A copy of a content uri that could not be read completely is not used as the file to upload. - onStopped() keeps the parts on the server when the system or the network stopped the work. The parts are removed only when the user cancelled it. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
Remove the rejected parts before marking the upload as restarted: when the DELETE fails, the retry must still be allowed to remove them. Keep the upload workspace in noBackupFilesDir instead of cacheDir. The system may clear the cache under storage pressure while an upload waits for the network. A new copy of the file changes its resume key and leaves the parts on the server orphaned, and after a process restart the content URI may no longer be readable. Finished workspaces are still removed by the worker. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-opus-5-5
A stop that came after the last check of doUpload() and before the chunked uploader was assigned found no uploader in onStopped(), so the upload ran on to the end. Assign the uploader first and check for a stop after that: an earlier stop is seen by the check, a later one reaches the uploader. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-opus-5-5
doWork() removes the cancel marker of its reference id when it ends, but WorkManager may call onStopped() after that. onStopped() then saw no cancel and only stopped the uploader, so the parts stayed on the server. Latch the cancel once it is seen, and check it once more right before the marker is removed. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-opus-5-5
- Delete the workspace of a cancelled run even when it was marked to be kept. - Copy a file:// source into the workspace, so the prepared upload does not depend on the caller's cache file. - Fix the upload folder before the uploader is published, so a cancel that arrives early can still remove the parts; abort at once if it already came. - Restart after HTTP 400 only when the assembly was rejected, not a part. - Sort the listed parts before looking for gaps. - Treat 401, 403, 408 and 429 while listing the parts as errors, not as an empty folder. - Delete a partial copy when the stream breaks off. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
copyFileToCache returned any existing target, including the leftover of a copy that died half way, and removed the target only on IOException. - Copy into a temporary file next to the target and rename it after a complete copy; remove the temporary file on any exception. - Treat an empty existing target as no cache hit and copy again. Assisted-by: Claude-Code:claude-sonnet-5-5 Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru>
stop() cancels the calls of the uploader to interrupt a request that is waiting on a dead network. The uploader built its client with okHttpClient.newBuilder(), which shares the Dispatcher of the application client. Cancelling all calls of that dispatcher would also cancel the chat requests, the signaling websocket and the other uploads. The uploader now builds its client with a Dispatcher of its own, so stop() and abortUpload() cancel only its own calls. Also: - abortUpload(localFile) removes the parts of an earlier run of a file for an uploader that has not started an upload, e.g. when the user cancels while the work waits for a retry. - deleteUploadedFile(path) removes a file that was already assembled on the server, e.g. when the user cancels between the upload and the share. - abortUpload() catches every exception: it runs inside the WorkManager cancellation handler, where any exception crashes the process. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
… it by the kind of its error A file upload stopped when the screen went off or the network was lost, and the worker gave up after four attempts, whatever the reason of the stop. Foreground service - The worker runs as a foreground work of type dataSync with an ongoing notification that shows the progress and a Cancel action. The manifest gets the FOREGROUND_SERVICE_DATA_SYNC permission, the dataSync type of the WorkManager SystemForegroundService and a non-exported cancel receiver. When the system refuses to start the service from the background (Android 12+), the upload continues as a regular work. Retry by the kind of the error (UploadRetryPolicy) - A lost network, a dropped TLS connection, 5xx (except 507), 408 and 429 are retried without a limit and do not count as an attempt: the work waits for the network and resumes from the parts on the server. - Only a refusal of the server (4xx, 507, certificate errors) counts, up to four times. The count is kept in the workspace. - A work that ends for good removes its parts from the server and shows the failed notification. It ends as success, because a failed work makes WorkManager fail the uploads queued behind it in the same conversation. - Backoff is linear. Exponential backoff with an unlimited number of attempts grows to hours. - The upload of a small file throws its HTTP errors, so the same policy applies to it. Stages and cancel - The stages "uploaded" and "shared", the id of the upload and the cancel flag are kept in the workspace. A run after a retry or a stop does not upload the file or share it a second time. - The user's cancel (chat or notification) only sets a flag in the workspace of the work. The worker sees it between its stages and in the progress callback, interrupts the upload, removes the parts, the assembled file and the placeholder message. WorkManager is not asked to cancel the work: a cancelled work in the queue of a conversation makes the next APPEND_OR_REPLACE delete the other works of that queue, even running ones. The work is found by its id or, after a restart of the app, by its tag. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
The cancel flag is the only way to stop an upload. When the user taps cancel and leaves the chat at once, the cleared ViewModel could drop the viewModelScope jobs before they set the flag and removed the placeholder, so the cancelled file would still be shared. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-opus-5-5
A stop that came after the last check and before the chunked uploader was assigned found no uploader in onStopped(), so the upload ran on to the end. Assign the uploader first and check for a stop after that: an earlier stop is seen by the check, a later one reaches the uploader. A cancelled work now finds the parts by the prepared file, so they are removed also when the uploader was assigned but did not start. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-opus-5-5
…d it The uploads of a conversation are chained with APPEND_OR_REPLACE, and WorkManager gives every request chained behind a cancelled or failed one the same state without running it. The new test runs three uploads of one conversation on a real WorkManager and cancels the first, a middle, the last and every one of them, and one by the reference tag as after a restart of the app. The other uploads must finish as SUCCEEDED and their placeholder rows must not stay PENDING. doWork() injects its dependencies only when none are set yet, so the test can give the worker a factory with its own dependencies. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
…ads behind it The cancel of an upload ends the work as success, because a failed work makes WorkManager fail every upload queued behind it in the conversation without running it. cancelledResult() removed the placeholder row without a guard: an exception from the database (e.g. a closed database) left doWork() as an exception, the work ended as FAILED, and the uploads behind the cancelled one ended as FAILED with their rows PENDING for ever. The row is removed by the chat as well, so a failed removal is only logged and the work still ends as success. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
…led run WorkManager ignores the result of a work it stopped (for example when the network is lost) and runs the work again. A user's cancel that came while the stopped run was still winding down made cancelledResult() end the run and the finally block of doWork() delete the workspace together with the cancel flag. The next run did not see the flag, uploaded the file and shared it, although its placeholder was already removed. cancelledResult() keeps the workspace when the run was stopped, as successResult() does. The next run finds the flag, removes the parts and the placeholder again (both are safe to repeat) and deletes the workspace. The same holds for the run that gets a retry because an older run still holds the lock of the workspace. The test stops the running work as WorkManager does, cancels it while the old run is still busy and checks that the flag survives and that the next run is cancelled too. The test restores the application it replaced. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
Answers of 5xx (except 507), 408 and 429 were retried without a limit. They are now a failure kind of their own, UNAVAILABLE, which is retried for 24 hours. The 24 hours count from the first "unavailable" answer after which the upload did not move on. The workspace keeps that time together with the progress it had then; a later answer with more progress (more chunks sent, or the file uploaded) starts the time again. Waiting for the network without an answer of the server is still not limited. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
ChunkFromFileRequestBody wraps every error of reading a chunk in a FileNotFoundException, with the real IOException as its cause. classify saw that cause as a lost network and retried for ever. A FileNotFoundException anywhere in the chain is now OTHER; an HTTP code still wins. The file is in noBackupFilesDir, so waiting does not repair it. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
A failed upload ends as SUCCEEDED on purpose, because WorkManager fails the works chained behind a failed one without running them. An observer of the work could therefore not tell a failure from a delivery, and a placeholder message of an upload that failed before it was inserted stayed PENDING. failUpload now sets KEY_UPLOAD_FAILED in the output data. A cancel does not. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
…oaded in chunks startForeground() ran before the preparation for every upload, so a 10 KB photo flashed a progress notification. The worker now asks for the size of the source first (File.length() for file://, FileUtils.querySize otherwise) and starts the foreground service only for files above the chunk threshold or of unknown size. An upload whose file is already on the server has one request left and needs no service either. The progress callback already checks isForeground. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
A failed upload removed the parts of a chunked upload but left the assembled file in the draft folder of the conversation. The server moves that file out of the draft folder only when the message is posted, so after a failure nobody claims it. failUpload now deletes it; a file the server already moved answers 404, which deleteUploadedFile swallows. The workspace remembers that the file went to the draft folder, so a failure that comes in a later run knows it too. A file in the attachment folder of Talk (servers without conversation subfolders) is a file of the user and stays. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
…red copy A later run of an upload asked the provider of the source for the size again, an IPC call that can be slow or answer differently, while the copy in the workspace is what is uploaded. The foreground service is now decided on the size of the prepared copy when there is one; the source is asked only before the copy exists. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
The progress for the 24 hours lived in memory of one run, so a run that began after a long pause counted from -1 and failed an upload the server had been taking data from. Now: - the furthest progress is stored in the workspace (written under the progress throttle, only when it grows), and the point of the clock is computed by the workspace: the stored maximum, or "uploaded" once the file is on the server; - the upload fails only after 24 hours AND at least 10 "unavailable" answers, so time without a network does not count as a refusing server; - a clock that went back starts the time again; - a workspace that cannot be written no longer throws out of the work (that failed the work and the uploads queued behind it): the error is logged and the upload is tried again. KEY_UPLOAD_FAILED now says to read the flag from the output data only, because the queue copies the output into the input of the next work. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
…eginning After the assembly was rejected (markRestarted) or the prepared file was made again (a new md5, so new chunks), the stored furthest progress, say 100, stayed. The new send goes 0..99 and never passes it, so every "unavailable" answer counted as no progress and the upload failed after 24 hours although the server took data. Both cases are a new send: the progress and the clock of the unavailable answers are removed. Signed-off-by: Krainov Gleb <krajnov.g@kontentplus.ru> Assisted-by: Claude-Code:claude-sonnet-5-5
69bbfe4 to
ee4c2b7
Compare
|
Thanks for the review. All points are addressed in new commits; both branches are rebased onto master with a force-push. Ceiling for the retries (69aa7bd, 4d48c3c, ee4c2b7) Local read error of the prepared file (0bcba5d)
Each fix has a unit test that fails without it. Unit tests, ktlint and detekt pass locally; detekt count unchanged. #6827 and this PR are rebased onto master in that order. This reply was drafted with AI assistance (Claude Code). |
Depends on #6827 (which depends on #6825). Closes #6826. Fixes #4115.
Problem
A file upload stops when the screen goes off or the network is lost. On current
master:UploadAndShareFilesWorkerruns withoutsetForeground, so the system can stop it when the screen is off.IOExceptionuses one ofMAX_UPLOAD_ATTEMPTS = 4, and a stop by a work constraint counts as an attempt. After that the upload is marked as failed, however long the network was gone.WorkManager.cancelWorkById(). A cancelled work that stays in the queue of a conversation (APPEND_OR_REPLACE) makes the nextenqueueUniqueWorkdelete the other works of that queue, including running ones.This PR builds on #6827, which makes a retry resume from the parts already on the server.
Solution
Foreground service
dataSyncwith an ongoing notification with the progress (updated once a second) and a Cancel action. The manifest getsFOREGROUND_SERVICE_DATA_SYNC, thedataSynctype for WorkManager'sSystemForegroundServiceand a non-exportedUploadCancelReceiver.Retry by the kind of the error (
UploadRetryPolicy, a pure class)success, notfailure, so WorkManager does not fail the uploads queued behind it.LINEAR: with an unlimited number of attemptsEXPONENTIALgrows to hours.FileUploaderthrows HTTP errors, so the policy applies to small files too.Cancel and stages
WorkManageris not asked to cancel. The worker checks the flag between its stages and in the progress callback, interrupts the upload, removes the parts, the assembled file and the placeholder message. The work is found by its id or, after a restart of the app, by a tag with the reference id.ChunkedFileUploaderDispatcher.okHttpClient.newBuilder()shares the dispatcher of the application client, socancelAll()instop()would also cancel chat requests and the signaling websocket. With its own dispatcherstop()interrupts only the requests of this uploader.abortUpload(localFile)(removes the parts of an earlier run for an uploader that did not start) anddeleteUploadedFile(path)(removes a file already assembled, when the user cancels before the share).Known limits
dataSyncforeground services on newer Android versions were not tested.Tests
Run:
UploadRetryPolicyTest: classification of network, TLS, 5xx, 408, 429, 507, 4xx and certificate errors (also when wrapped by RxJava) and the decision (retry without limit, limit of four for refusals).UploadWorkspaceTest: stages, upload id, cancel flag and server error count survive a new preparation.FileUploaderTest: HTTP errors of a small upload are thrown.ChunkedFileUploaderResumeTest:stop()does not cancel calls of other clients built from the same base client,abortUpload(localFile),deleteUploadedFile().UploadProgressThrottleTest: the notification is updated at most once a second.The worker itself has no instrumented test.
testGplayDebugUnitTestpasses (818 tests, 0 failures, 1 skipped), ktlint is clean. detekt reports no new finding againstmaster; the existingLargeClassfinding ofChatViewModelgrows from 1748 to 1749 lines (one line incancelUpload).Tested by the author on a Huawei DEL-LX9 with a fork build that contains #6825, #6827 and this PR: large file, screen off, airplane mode — the upload resumed and completed.
🏁 Checklist
/backport to stable-xx.x🤖 AI (if applicable)
The analysis, the code and the tests were prepared with an AI assistant (Claude Code). The commits carry an
Assisted-bytrailer. I reviewed the change and tested it on a device.🤖 Generated with Claude Code