mirror of
https://github.com/alexta69/metube.git
synced 2026-09-21 21:45:04 +00:00
Alex Shnitman
c9c507f939
PersistentQueue.put/delete wrote the whole queue inline: serialize, write a temp file, fsync it, rename, then fsync the directory. All of that ran synchronously inside async callers, so on a slow or contended filesystem a single queue mutation stalled every other request for as long as the two fsyncs took. Adds and completions are exactly when it fires, which matches the reported "hiccups happen when something is pushing into the queue". put/delete are now coroutines. The payload is still serialized on the event loop -- it is pure CPU and sub-millisecond -- and only the write goes to a thread, so the writer never walks live DownloadInfo objects while the loop mutates them. Each queue gets its own single-worker executor rather than sharing the default one, because extract_info can hold default-executor threads for minutes and would leave state writes queued behind exactly when they are needed. Awaiting the write makes interleaving possible where it was not before, so a lock now covers the mutate-write-rollback section: the invariant that in-memory state never diverges from what is on disk is unchanged, including the rollback when a write fails. On shutdown the queues are drained rather than cancelled, so a write in flight still lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>