Skip to content

Conversation

@emmiegit
Copy link
Member

@emmiegit emmiegit commented Aug 31, 2025

With the two-step upload process now (create a pending upload, and use the uploaded object to add a file revision), we now leave behind blob_pending rows which need cleaning up after they expire.

This PR adds this recurring job, which deletes the object from the S3 bucket (if it exists), and then deletes the row from the database.

I also use this opportunity to ensure these recurring jobs are queued to rsmq on setup. It is done during queue creation to avoid the job being added multiple times, for each start.

Logs from local test
deepwell-1   | deepwell::services::blob::service Pruning expired pending blobs from database and S3
deepwell-1   | sqlx::query summary="SELECT \"blob_pending\".\"external_id\", \"blob_pending\".\"s3_path\" FROM …" db.statement="\n\nSELECT \"blob_pending\".\"external_id\", \"blob_pending\".\"s3_path\" FROM \"blob_pending\" WHERE \"blob_pending\".\"expires_at\" <= $1\n" rows_affected=1 rows_returned=1 elapsed=4.29537ms elapsed_secs=0.00429537
deepwell-1   | sqlx::query summary="DELETE FROM \"blob_pending\" WHERE …" db.statement="\n\nDELETE FROM \"blob_pending\" WHERE \"blob_pending\".\"external_id\" IN ($1)\n" rows_affected=1 rows_returned=0 elapsed=954.105µs elapsed_secs=0.000954105
deepwell-1   | deepwell::services::job::service Queuing job PrunePendingUploads (delay Some(600s))

@emmiegit emmiegit self-assigned this Aug 31, 2025
@emmiegit emmiegit force-pushed the WJ-1225-prune-expired branch from 3e60c28 to 3ebfe30 Compare August 31, 2025 23:04
Previous one (for an object hash) was unused), this one is just for
bucket existence.
@emmiegit emmiegit force-pushed the WJ-1225-prune-expired branch from 3ebfe30 to 4244c9d Compare August 31, 2025 23:08
@emmiegit emmiegit force-pushed the WJ-1225-prune-expired branch from 4244c9d to df20534 Compare August 31, 2025 23:22
@emmiegit emmiegit marked this pull request as ready for review August 31, 2025 23:23
@emmiegit emmiegit merged commit dcaf7c1 into develop Sep 1, 2025
8 checks passed
@emmiegit emmiegit deleted the WJ-1225-prune-expired branch September 1, 2025 00:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants