A download waiting for a MAX_CONCURRENT_DOWNLOADS slot sat at status
'pending' — the same status as an item added with auto-start off, which
is waiting for the user to press Start. The Downloading table draws its
Start button for exactly that status, so every queued row offered one.
Pressing it did nothing. start_pending() looks the id up in self.pending
first, and a queued download is not there; the fallback branch only acts
on 'scheduled' items, so the call fell through and still returned
{'status': 'ok'} — the UI reported success for a no-op.
One status name was covering two different states. A download in
self.queue waiting on the semaphore is now 'queued', leaving 'pending'
to mean only "waiting for you to press Start". The template condition is
unchanged and now excludes these rows by construction, and a 'Queued'
badge says why the row is idle instead of leaving a bare empty bar.
start_pending() notifies on promotion as well: with the slots saturated
the wait before Download.start() reports 'preparing' is unbounded, and
until something lands the client keeps showing the button it outgrew.
Persisted state needs no migration — __import_queue re-adds saved items
through __add_download, which sets the new status.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
validate_url resolved every submitted hostname in the server process before
yt-dlp saw it. Behind a proxy that does its own DNS -- an HTTP proxy, socks5h,
socks4a, or the plain socks5 yt-dlp rewrites to socks5h -- that lookup is both
wrong and harmful: it describes this host's network rather than the proxy's,
and it leaks the hostname of every queued URL to the local resolver, which is
the one thing a SOCKS/Tor setup exists to prevent. It also failed closed when
only the proxy could resolve the name, so a container pointed at the proxy's
DNS port refused every add with 'Could not resolve host'.
The address check is now skipped for hostnames that the carrying proxy will
resolve, and kept everywhere else: for direct fetches, for hosts excluded by
no_proxy, for socks4 (which resolves locally), and for hosts written as IP
literals, which need no lookup and leak nothing. Scheme validation, the
localhost/metadata blocklist and the connect-time socket guard are unchanged.
download_proxies mirrors YoutubeDL.proxies rather than importing it: that
property is only reachable from a constructed instance, and since it decides
whether a security check runs, a quiet upstream change should leave the check
in place rather than silently skip it.
Also makes ALLOW_PRIVATE_ADDRESSES explicit in the download-queue test config
-- unset on a MagicMock it is truthy, which had validate_url bypassing every
check those tests asked it to run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PersistentQueue.put/delete wrote the whole queue inline: serialize, write
a temp file, fsync it, rename, then fsync the directory. All of that ran
synchronously inside async callers, so on a slow or contended filesystem
a single queue mutation stalled every other request for as long as the
two fsyncs took. Adds and completions are exactly when it fires, which
matches the reported "hiccups happen when something is pushing into the
queue".
put/delete are now coroutines. The payload is still serialized on the
event loop -- it is pure CPU and sub-millisecond -- and only the write
goes to a thread, so the writer never walks live DownloadInfo objects
while the loop mutates them. Each queue gets its own single-worker
executor rather than sharing the default one, because extract_info can
hold default-executor threads for minutes and would leave state writes
queued behind exactly when they are needed.
Awaiting the write makes interleaving possible where it was not before,
so a lock now covers the mutate-write-rollback section: the invariant
that in-memory state never diverges from what is on disk is unchanged,
including the rollback when a write fails. On shutdown the queues are
drained rather than cancelled, so a write in flight still lands.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A "Remove sponsor segments" switch next to "Split by chapters" queues
the download with the same postprocessor pair the CLI's
--sponsorblock-remove sponsor builds (SponsorBlock + ModifyChapters).
The flag persists as a cookie like the other form options, survives in
the queue records, and is carried into retries.
The pair is registered above the chapter-splitting block: yt-dlp runs
same-stage postprocessors in list order, so ModifyChapters has to
rewrite the chapter list before FFmpegSplitChapters cuts the file up,
matching what the CLI builds for --sponsorblock-remove sponsor
--split-chapters. With both toggles on the other way around the chapter
files keep the sponsor segments and the removal desyncs the remaining
chapter timings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
__is_channel_extraction keyed on id == channel_id. That holds for a channel
tab - /channel/UC..., and the videos, streams, shorts and playlists tabs of a
/@handle URL - but not for a channel addressed on its own: yt-dlp reports the
id in the form the channel was asked for, so a bare handle URL yields '@handle'
and a legacy /c/ URL yields the vanity name.
Neither matched, so both fell through to OUTPUT_TEMPLATE_PLAYLIST and the
folder came out as the feed's title. That is why e2c7778 fixed the reporter's
tab URLs while a bare channel URL - what you get copying the address bar - went
on ignoring OUTPUT_TEMPLATE.
Both forms match uploader_id, which is the handle either way, so compare
against that as well, without case: a legacy vanity name and the handle it
became need not agree on it. A real playlist carries its owner's channel_id and
uploader_id but keeps an id of its own, so it still reads as a playlist; no
playlist id can collide with a handle, since those are 'PL...', 'OLAK...' and
the like.
Verified against the live extractor for all six channel URL forms and a real
playlist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
yt-dlp emits the feed-level .info.json, description and thumbnail from
__process_playlist_result without consulting `download`, so they fell out of
MeTube's classification pass with nothing steering them: they landed in
DOWNLOAD_DIR under yt-dlp's own pl_* names, ignoring OUTPUT_TEMPLATE, the
download's folder and AUDIO_DOWNLOAD_DIR. #1040 reported the names; #660 asked
for control over them. Same accident from both sides.
The classification pass now writes nothing at all, and the files are produced
once the feed has been accepted and its type is known, reusing the template its
items use — OUTPUT_TEMPLATE_CHANNEL for a channel, OUTPUT_TEMPLATE_PLAYLIST for
a playlist, falling back to OUTPUT_TEMPLATE — evaluated against the feed dict.
With the defaults that puts them in the same folder as the videos, named after
the feed, which is the Jellyfin layout #660 asked for. No new environment
variable: knowing the feed type is what makes reusing the item template
possible, and doing this after extraction is what makes the type known.
The write re-runs yt-dlp over a copy of the feed with its entries removed,
which reaches the playlist-file writing without re-extracting anything and
without touching yt-dlp's private write helpers. It runs in an executor and
never fails the add.
Nothing new appears for anyone who hasn't enabled writeinfojson /
writethumbnail, an explicit allow_playlist_files=false still turns it off, and
a failed or cancelled add no longer leaves metadata behind.
Subscription scans keep allow_playlist_files=False: a scan is a timer-driven
poll, and items it finds are queued through the download queue, which does the
writing. Previously every check interval rewrote these files.
Test doubles for __extract_info now take *args/**kwargs.
Two review fixes on top of the retry endpoint:
- __add_entry dropped retry_entry when extraction returned an unprocessed
url/url_transparent result and it recursed back into add(). Since
__extract_info runs with extract_flat=True, that path is live, and a retried
playlist item taking it fell back to OUTPUT_TEMPLATE and landed in the root
directory instead of its playlist folder. The playlist child loop keeps
passing retry_entry=None on purpose: those entries get fresh playlist context
stamped on them from the current extraction.
- retry() called dqueue.add() directly, so it bypassed the
parse_download_options gates that /add applies. Stored ytdl_options_overrides
were re-applied even after ALLOW_YTDL_OPTIONS_OVERRIDES was turned off, and
preset names removed from the configuration were still passed through. Both
are re-checked against the current configuration at retry time.
Adds __record_add_failure so a URL that fails before a real download starts
(unsupported/unextractable URL, SSRF-rejected, extraction error) appears in the
Completed list as a red-cross entry with retry and error-detail, instead of only
a transient toast and a server log line. Keyed by info.url like any errored
download. Includes _short_title_for_failed_url for a readable hostname title.
The frontend hunk removing the 'Click for details' hint from every error row was
dropped from this merge; that affordance (added in fd3aaea, #143) is kept.
Co-authored-by: streamer1122 <streamer1122@users.noreply.github.com>
Addresses a full-project review. Backend correctness and availability:
- ytdl: cancel() only SIGKILLs the child's process group when the child
actually became its own group leader, so a race (or failed setpgrp)
can no longer kill the whole server; kill the group on cancel and on
shutdown to avoid orphaned ffmpeg children
- ytdl: dedicated ThreadPoolExecutor for download supervision so active
downloads can't starve extract_info / live probes on the default pool
- ytdl/main/subscriptions: route fire-and-forget tasks through a
bg_tasks helper that keeps a strong ref and logs failures
- subscriptions: run flat-playlist extraction in an executor and check
feeds with bounded concurrency so one slow feed can't block the loop;
set last_checked on failure so broken feeds aren't retried every 60s
- main: validate ids on /start & /delete and numeric env vars at startup;
return 400 (not 500) on bad subscriptions/update input; serve /history
from memory; move get_custom_dirs off the event loop; restrict t=
stripping to YouTube hosts; drop double percent-decode in state guard
- dl_formats/ytdl: enforce requested caption format via
FFmpegSubtitlesConvertor and strip VTT header metadata only in the
pre-cue region so real dialogue is preserved
- ytdl: throttle progress events, dedup adds against pending, clear
filename/size on error and reject out-of-dir trashcan deletes, pin
fork start-method on Linux only
Frontend:
- retry deletes the done record only after a successful re-add
- surface HTTP errors for delete/start and reset the deleting flag
- ignore late 'updated' events for rows no longer in the queue
- track table rows by map key; FileSizePipe uses base-1024
Also: HTTPS-aware Docker healthcheck, dead-code removal, and shared
helpers for path-containment and yt-dlp option merging. Adds/updates
unit tests throughout (250 backend tests passing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Detect YouTube channel tabs that yt-dlp reports as playlists so channel downloads use OUTPUT_TEMPLATE_CHANNEL and its empty fallback instead of OUTPUT_TEMPLATE_PLAYLIST.
Alex Shnitman