yt-dlp documents 'filename' as always present in a progress hook, so the
update_status branch that stats it ran on every forwarded tick: throttled
to one every 0.5s per download, times MAX_CONCURRENT_DOWNLOADS. Those are
blocking syscalls on the event loop, and when the filesystem is slow each
one freezes every other request the server is serving -- which is what a
bare GET timing out at >10s looks like from outside.
The call was also useless while it was expensive. Until the download
finishes the bytes live in tmpfilename; 'filename' is the destination,
which does not exist yet, so os.path.exists() returned False and size
stayed None. It only yields a real value on a terminal status, and the
Downloading table has no size column, so nothing displayed it before
completion either way.
Stat only when the status is 'finished'. That covers both moments a file
genuinely exists at that path: yt-dlp's own finished status, and the
MoveFiles postprocessor reporting the final merged name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a "Remove sponsor segments" switch to the shared options panel,
building the same SponsorBlock + ModifyChapters pair the CLI's
--sponsorblock-remove sponsor does, and carries the flag through
subscriptions so the panel's control applies to both forms.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The image ships yt-dlp's bgutil PO token provider and starts it on
loopback, where the plugin dials it at http://127.0.0.1:4416. Since
482381d scoped the connect-time allowance to the configured proxy, the
download subprocess could no longer resolve it:
Refusing to connect to non-global address for host '127.0.0.1'
which surfaces as the plugin's "Error reaching GET .../ping". Metadata
extraction runs in the main process and installs no guard, so titles kept
resolving while the download itself ran without a token — and YouTube
increasingly answers those with 403.
The allowance already had the right shape for this; it was just named for
its only user. Endpoints the operator or the image configured are now
allowed as a class: install_socket_guard takes service_urls alongside
proxy_urls, and ytdl derives them from the bundled default plus any
base_url set through the youtubepot-bgutilhttp (or the deprecated youtube
getpot_bgutil_baseurl) extractor argument. The bundled server runs either
way, so it stays allowed when a base URL is configured.
Matching stays exact host:port on the configured string, so nothing else
on loopback opens up: a hostile media URL naming the endpoint reaches a
token server with two endpoints and nothing worth reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Subscriptions download unattended, which is where skipping sponsor reads
is most useful, so the flag now travels the same path the other download
options take: stored on SubscriptionInfo, persisted in the record, and
passed to add_entry for every entry a check queues.
Like the clip bounds, it is set when the subscription is created; the
update endpoint's field list is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A "Remove sponsor segments" switch next to "Split by chapters" queues
the download with the same postprocessor pair the CLI's
--sponsorblock-remove sponsor builds (SponsorBlock + ModifyChapters).
The flag persists as a cookie like the other form options, survives in
the queue records, and is carried into retries.
The pair is registered above the chapter-splitting block: yt-dlp runs
same-stage postprocessors in list order, so ModifyChapters has to
rewrite the chapter list before FFmpegSplitChapters cuts the file up,
matching what the CLI builds for --sponsorblock-remove sponsor
--split-chapters. With both toggles on the other way around the chapter
files keep the sponsor segments and the removal desyncs the remaining
chapter timings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
__is_channel_extraction keyed on id == channel_id. That holds for a channel
tab - /channel/UC..., and the videos, streams, shorts and playlists tabs of a
/@handle URL - but not for a channel addressed on its own: yt-dlp reports the
id in the form the channel was asked for, so a bare handle URL yields '@handle'
and a legacy /c/ URL yields the vanity name.
Neither matched, so both fell through to OUTPUT_TEMPLATE_PLAYLIST and the
folder came out as the feed's title. That is why e2c7778 fixed the reporter's
tab URLs while a bare channel URL - what you get copying the address bar - went
on ignoring OUTPUT_TEMPLATE.
Both forms match uploader_id, which is the handle either way, so compare
against that as well, without case: a legacy vanity name and the handle it
became need not agree on it. A real playlist carries its owner's channel_id and
uploader_id but keeps an id of its own, so it still reads as a playlist; no
playlist id can collide with a handle, since those are 'PL...', 'OLAK...' and
the like.
Verified against the live extractor for all six channel URL forms and a real
playlist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MeTube never sets ignoreerrors, so yt-dlp's trouble() raises DownloadError
instead of returning nonzero for failures like "No video formats found!".
That is the path issue #1047 reports, and it dropped the warnings that
explain the failure because only the nonzero-return branch attached them.
The exception branch now reports the retained warnings with the exception
text as the final line, so the actual error stays prominent under the
context. yt-dlp labels errors "ERROR:" but hands warnings to the logger
unlabelled, so a last warning that repeats the error text is skipped.
Retention is bounded to the last five distinct warnings: fragmented and
live downloads emit a warning per fragment, and the joined message is
persisted with the completed queue and broadcast to every client.
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Sites that put a description in the title produce output names past what
the filesystem accepts, and the download fails outright with
"[Errno 36] File name too long". The name is now shortened to fit inside
prepare_filename, which every output path already passes through, so the
main file, its chapter files, thumbnails and subtitles stay consistent.
The limit is read from the filesystem (PC_NAME_MAX, falling back to 255)
and counted in bytes, not characters: a title of accented or CJK
characters reaches it in a third of the characters. The extension is
kept, a cut landing inside a multi-byte character does not leave a broken
sequence, and a reserve is held back for the suffixes yt-dlp appends
afterwards -- '.part', '.ytdl', '.f<format_id>', '-Frag<n>' -- since it
is those that pushed the reported name over the limit.
Names that already fit are untouched, so nothing that downloads today
changes name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scoping the connect-time allowance to the configured proxy kept the exception
tied to loopback, so a proxy anywhere else internal — the common case of a
socks5 or HTTP proxy on the LAN — was refused with "Refusing to connect to
non-global address". The only workaround was ALLOW_PRIVATE_ADDRESSES, which
switches the whole guard off, a far larger concession than the setup needs.
The allowance was never really about loopback: it is about the operator having
named this host:port as a proxy. Widen it to any address at a configured proxy
endpoint and nothing is given away, because the match is on the configured host
string rather than the resolved address — a hostile media URL that resolves to
the proxy's address under another name gets no allowance, and one that names the
proxy endpoint itself only reaches the proxy. Every other internal destination
stays refused.
Also log each configured proxy endpoint, so the next report of this shape can be
diagnosed from the log rather than from the guard's source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The connect-time guard permitted every loopback address so that a locally
configured proxy (proxy: http://127.0.0.1:9050) stayed reachable. But the
connect guard is the only check that media URLs derived from remote metadata
ever face — validate_url sees just the submitted URL — so that blanket
allowance let a remote manifest steer the download subprocess at services
bound to the server's loopback interface, with the response written to the
download directory and served back by the UI.
Permit loopback only at the host:port of a proxy the operator configured,
taken from yt-dlp's proxy option and the *_proxy environment variables.
Nothing is lost: when a proxy is in use yt-dlp hands it the media URL instead
of resolving that URL locally, so the two cases never overlap. Every other
loopback destination now falls under the same is_global policy as the rest.
Reported by m3rl1nu5 (https://github.com/hai135) in GHSA-73g4-qhhq-c32c.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
yt-dlp emits the feed-level .info.json, description and thumbnail from
__process_playlist_result without consulting `download`, so they fell out of
MeTube's classification pass with nothing steering them: they landed in
DOWNLOAD_DIR under yt-dlp's own pl_* names, ignoring OUTPUT_TEMPLATE, the
download's folder and AUDIO_DOWNLOAD_DIR. #1040 reported the names; #660 asked
for control over them. Same accident from both sides.
The classification pass now writes nothing at all, and the files are produced
once the feed has been accepted and its type is known, reusing the template its
items use — OUTPUT_TEMPLATE_CHANNEL for a channel, OUTPUT_TEMPLATE_PLAYLIST for
a playlist, falling back to OUTPUT_TEMPLATE — evaluated against the feed dict.
With the defaults that puts them in the same folder as the videos, named after
the feed, which is the Jellyfin layout #660 asked for. No new environment
variable: knowing the feed type is what makes reusing the item template
possible, and doing this after extraction is what makes the type known.
The write re-runs yt-dlp over a copy of the feed with its entries removed,
which reaches the playlist-file writing without re-extracting anything and
without touching yt-dlp's private write helpers. It runs in an executor and
never fails the add.
Nothing new appears for anyone who hasn't enabled writeinfojson /
writethumbnail, an explicit allow_playlist_files=false still turns it off, and
a failed or cancelled add no longer leaves metadata behind.
Subscription scans keep allow_playlist_files=False: a scan is a timer-driven
poll, and items it finds are queued through the download queue, which does the
writing. Previously every check interval rewrote these files.
Test doubles for __extract_info now take *args/**kwargs.
Two review fixes on top of the retry endpoint:
- __add_entry dropped retry_entry when extraction returned an unprocessed
url/url_transparent result and it recursed back into add(). Since
__extract_info runs with extract_flat=True, that path is live, and a retried
playlist item taking it fell back to OUTPUT_TEMPLATE and landed in the root
directory instead of its playlist folder. The playlist child loop keeps
passing retry_entry=None on purpose: those entries get fresh playlist context
stamped on them from the current extraction.
- retry() called dqueue.add() directly, so it bypassed the
parse_download_options gates that /add applies. Stored ytdl_options_overrides
were re-applied even after ALLOW_YTDL_OPTIONS_OVERRIDES was turned off, and
preset names removed from the configuration were still passed through. Both
are re-checked against the current configuration at retry time.
Added track_number to the set of keys preserved when compacting persisted playlist entries, ensuring this metadata is retained for accurate track ordering and display.
Adds __record_add_failure so a URL that fails before a real download starts
(unsupported/unextractable URL, SSRF-rejected, extraction error) appears in the
Completed list as a red-cross entry with retry and error-detail, instead of only
a transient toast and a server log line. Keyed by info.url like any errored
download. Includes _short_title_for_failed_url for a readable hostname title.
The frontend hunk removing the 'Click for details' hint from every error row was
dropped from this merge; that affordance (added in fd3aaea, #143) is kept.
Co-authored-by: streamer1122 <streamer1122@users.noreply.github.com>
Adds MusicMetadataPreProcessor (app/music_metadata.py), a pre_process
postprocessor that enriches the info dict using only extractor-owned fields:
track-number/total resolution, album-title fallback, and square-thumbnail
preference. Track numbers and album flow into files through yt-dlp's existing
FFmpegMetadata embedding, so no per-format tag-writing code and no new
dependency. The earlier mutagen writer and its download-failing
PostProcessingError path were dropped per review.
Co-authored-by: jahruz67 <jahruz67@users.noreply.github.com>
Wraps Download.close() in try/finally and nulls self.status_queue so the
per-download manager.Queue() proxy is released once the completed Download is
retained in the done list. Previously every finished download permanently
pinned one Manager-process connection, accumulating file descriptors until
the instance hit 'too many open files' and self-terminated (#485, #980).
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
The SSRF guard rejects any address that isn't globally routable, which breaks
proxy/VPN setups that resolve hosts into private or special-use ranges. The
reported case is Fake-IP clients (sing-box, Clash, Mihomo) that map YouTube to
the RFC 2544 benchmarking range 198.18.0.0/15 to tunnel the traffic; MeTube
rejected it with "Refusing to fetch internal address" even though yt-dlp itself
handles it fine.
Add a boolean ALLOW_PRIVATE_ADDRESSES (default false) that, when set, skips both
layers: validate_url returns after scheme validation without the internal-host
checks, and the connect-time socket guard is not installed. Threaded to the
download subprocess via Download so the guard sees it too. Scheme validation
(http/https only) still applies. Documented in the README as a trust-your-network
opt-out that disables SSRF protection.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The connect-time getaddrinfo guard added for redirect/rebinding SSRF covers
only the download subprocess: metadata extraction runs in the main process
(where a process-wide guard would reject the server's own HOST=0.0.0.0 bind),
and native curl_cffi/libcurl resolution used by --impersonate bypasses Python's
socket module. Record both in url_guard's docstring and at the extraction site
so the boundary is explicit; network isolation remains the backstop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
validate_url only inspects the submitted URL string. yt-dlp then follows HTTP
redirects and resolves media URLs from remote metadata without re-checking, so
an allowed URL that 302s to http://169.254.169.254/ (cloud metadata) or an
RFC1918 host is still fetched — the guard's own docstring scoped this out.
Install a getaddrinfo guard in the download subprocess that re-validates every
resolved address at actual connect time, covering redirects and DNS rebinding
for any backend resolving through Python's socket module (urllib, requests).
Loopback is permitted so locally-configured proxies keep working; link-local,
RFC1918 and unique-local are blocked. Native resolvers (curl_cffi/libcurl via
--impersonate) bypass this and rely on network isolation as the backstop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-download chapter_template was validated only against literal ".."
in the template string, but yt-dlp expands %(section_title)s (and every
other field) from attacker-controlled metadata at download time. On POSIX
hosts yt-dlp does not neutralise a ".." path component, so a chapter titled
".." turns a guard-passing template like
"%(section_title)s/%(section_title)s/x.%(ext)s" into "../../x.mp4" and writes
outside DOWNLOAD_DIR. The same class of escape applies to any multi-segment
output template (default/playlist/channel) whose fields resolve to "..".
The template string can never see the "..": it only exists after expansion.
So move the check to the one point every output path flows through —
YoutubeDL.prepare_filename — via a _ConfinedYoutubeDL subclass that refuses
any resolved path outside the download/temp roots (fail closed). This covers
the main file, split-chapter files, thumbnails and subtitles in one place.
With the chokepoint authoritative, the scattered ingress string checks
(chapter_template, custom_name_prefix) and the weaker _output_dir_escapes
literal-prefix check are removed. Tests move from the ingress layer to the
chokepoint, exercising the real metadata-resolution vector.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds _AlbumArtistPostProcessor, a yt-dlp pre_process postprocessor that
fills album_artist from the '<artist> - Topic' channel/uploader signal,
falling back to the first credited artist, when album metadata exists
but no album artist is set.
Closes#1025.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
User-submitted URLs were passed straight to yt-dlp's generic extractor,
letting the server fetch internal endpoints (cloud metadata, loopback,
RFC1918 hosts). Add a url_guard.validate_url check at every URL ingress
(add, subscribe, and nested playlist recursion) that rejects non-http(s)
schemes and hosts resolving to non-global addresses, while leaving bare
video IDs and search prefixes untouched.
Sanitize path separators and .. segments in playlist/channel titles before they are baked into yt-dlp output templates, and refuse downloads whose resolved output directory escapes DOWNLOAD_DIR.
Addresses a full-project review. Backend correctness and availability:
- ytdl: cancel() only SIGKILLs the child's process group when the child
actually became its own group leader, so a race (or failed setpgrp)
can no longer kill the whole server; kill the group on cancel and on
shutdown to avoid orphaned ffmpeg children
- ytdl: dedicated ThreadPoolExecutor for download supervision so active
downloads can't starve extract_info / live probes on the default pool
- ytdl/main/subscriptions: route fire-and-forget tasks through a
bg_tasks helper that keeps a strong ref and logs failures
- subscriptions: run flat-playlist extraction in an executor and check
feeds with bounded concurrency so one slow feed can't block the loop;
set last_checked on failure so broken feeds aren't retried every 60s
- main: validate ids on /start & /delete and numeric env vars at startup;
return 400 (not 500) on bad subscriptions/update input; serve /history
from memory; move get_custom_dirs off the event loop; restrict t=
stripping to YouTube hosts; drop double percent-decode in state guard
- dl_formats/ytdl: enforce requested caption format via
FFmpegSubtitlesConvertor and strip VTT header metadata only in the
pre-cue region so real dialogue is preserved
- ytdl: throttle progress events, dedup adds against pending, clear
filename/size on error and reject out-of-dir trashcan deletes, pin
fork start-method on Linux only
Frontend:
- retry deletes the done record only after a successful re-add
- surface HTTP errors for delete/start and reset the deleting flag
- ignore late 'updated' events for rows no longer in the queue
- track table rows by map key; FileSizePipe uses base-1024
Also: HTTPS-aware Docker healthcheck, dead-code removal, and shared
helpers for path-containment and yt-dlp option merging. Adds/updates
unit tests throughout (250 backend tests passing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Detect YouTube channel tabs that yt-dlp reports as playlists so channel downloads use OUTPUT_TEMPLATE_CHANNEL and its empty fallback instead of OUTPUT_TEMPLATE_PLAYLIST.
The playlist/channel processing loop now sets playlist_count,
playlist_autonumber, n_entries, and __last_playlist_index on each
video entry so that templates like %(playlist_autonumber)s,
%(playlist_count)s, and %(playlist_index&{} - |)s resolve correctly
instead of showing NA.
Also updates _compact_persisted_entry to preserve n_entries and
__last_playlist_index across restarts.
Fixes#692
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/b5aeb55a-3197-4a14-b8b4-96c9a67796e8
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
Replace the hand-rolled _outtmpl_substitute_field() / _compile_outtmpl_pattern()
with a new _resolve_outtmpl_fields() that delegates to yt-dlp's
YoutubeDL.evaluate_outtmpl(). This gives playlist/channel output templates
access to yt-dlp's full template syntax: defaults (%(field|fallback)s),
conditional formatting (%(field&prefix {})s), math (%(field+N)d),
datetime formatting (%(field>%Y-%m-%d)s), and more.
Only field references whose root name matches the targeted prefix (e.g.
"playlist" or "channel") are resolved; all other references remain as
template placeholders for yt-dlp to fill during the actual download.
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/0ae5ff34-540f-4fc8-a81c-358fb92b7c15
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
Alex Shnitman