yt-dlp emits the feed-level .info.json, description and thumbnail from
__process_playlist_result without consulting `download`, so they fell out of
MeTube's classification pass with nothing steering them: they landed in
DOWNLOAD_DIR under yt-dlp's own pl_* names, ignoring OUTPUT_TEMPLATE, the
download's folder and AUDIO_DOWNLOAD_DIR. #1040 reported the names; #660 asked
for control over them. Same accident from both sides.
The classification pass now writes nothing at all, and the files are produced
once the feed has been accepted and its type is known, reusing the template its
items use — OUTPUT_TEMPLATE_CHANNEL for a channel, OUTPUT_TEMPLATE_PLAYLIST for
a playlist, falling back to OUTPUT_TEMPLATE — evaluated against the feed dict.
With the defaults that puts them in the same folder as the videos, named after
the feed, which is the Jellyfin layout #660 asked for. No new environment
variable: knowing the feed type is what makes reusing the item template
possible, and doing this after extraction is what makes the type known.
The write re-runs yt-dlp over a copy of the feed with its entries removed,
which reaches the playlist-file writing without re-extracting anything and
without touching yt-dlp's private write helpers. It runs in an executor and
never fails the add.
Nothing new appears for anyone who hasn't enabled writeinfojson /
writethumbnail, an explicit allow_playlist_files=false still turns it off, and
a failed or cancelled add no longer leaves metadata behind.
Subscription scans keep allow_playlist_files=False: a scan is a timer-driven
poll, and items it finds are queued through the download queue, which does the
writing. Previously every check interval rewrote these files.
Test doubles for __extract_info now take *args/**kwargs.
Two review fixes on top of the retry endpoint:
- __add_entry dropped retry_entry when extraction returned an unprocessed
url/url_transparent result and it recursed back into add(). Since
__extract_info runs with extract_flat=True, that path is live, and a retried
playlist item taking it fell back to OUTPUT_TEMPLATE and landed in the root
directory instead of its playlist folder. The playlist child loop keeps
passing retry_entry=None on purpose: those entries get fresh playlist context
stamped on them from the current extraction.
- retry() called dqueue.add() directly, so it bypassed the
parse_download_options gates that /add applies. Stored ytdl_options_overrides
were re-applied even after ALLOW_YTDL_OPTIONS_OVERRIDES was turned off, and
preset names removed from the configuration were still passed through. Both
are re-checked against the current configuration at retry time.
Added track_number to the set of keys preserved when compacting persisted playlist entries, ensuring this metadata is retained for accurate track ordering and display.
Adds __record_add_failure so a URL that fails before a real download starts
(unsupported/unextractable URL, SSRF-rejected, extraction error) appears in the
Completed list as a red-cross entry with retry and error-detail, instead of only
a transient toast and a server log line. Keyed by info.url like any errored
download. Includes _short_title_for_failed_url for a readable hostname title.
The frontend hunk removing the 'Click for details' hint from every error row was
dropped from this merge; that affordance (added in fd3aaea, #143) is kept.
Co-authored-by: streamer1122 <streamer1122@users.noreply.github.com>
Adds MusicMetadataPreProcessor (app/music_metadata.py), a pre_process
postprocessor that enriches the info dict using only extractor-owned fields:
track-number/total resolution, album-title fallback, and square-thumbnail
preference. Track numbers and album flow into files through yt-dlp's existing
FFmpegMetadata embedding, so no per-format tag-writing code and no new
dependency. The earlier mutagen writer and its download-failing
PostProcessingError path were dropped per review.
Co-authored-by: jahruz67 <jahruz67@users.noreply.github.com>
Wraps Download.close() in try/finally and nulls self.status_queue so the
per-download manager.Queue() proxy is released once the completed Download is
retained in the done list. Previously every finished download permanently
pinned one Manager-process connection, accumulating file descriptors until
the instance hit 'too many open files' and self-terminated (#485, #980).
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
The SSRF guard rejects any address that isn't globally routable, which breaks
proxy/VPN setups that resolve hosts into private or special-use ranges. The
reported case is Fake-IP clients (sing-box, Clash, Mihomo) that map YouTube to
the RFC 2544 benchmarking range 198.18.0.0/15 to tunnel the traffic; MeTube
rejected it with "Refusing to fetch internal address" even though yt-dlp itself
handles it fine.
Add a boolean ALLOW_PRIVATE_ADDRESSES (default false) that, when set, skips both
layers: validate_url returns after scheme validation without the internal-host
checks, and the connect-time socket guard is not installed. Threaded to the
download subprocess via Download so the guard sees it too. Scheme validation
(http/https only) still applies. Documented in the README as a trust-your-network
opt-out that disables SSRF protection.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The connect-time getaddrinfo guard added for redirect/rebinding SSRF covers
only the download subprocess: metadata extraction runs in the main process
(where a process-wide guard would reject the server's own HOST=0.0.0.0 bind),
and native curl_cffi/libcurl resolution used by --impersonate bypasses Python's
socket module. Record both in url_guard's docstring and at the extraction site
so the boundary is explicit; network isolation remains the backstop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
validate_url only inspects the submitted URL string. yt-dlp then follows HTTP
redirects and resolves media URLs from remote metadata without re-checking, so
an allowed URL that 302s to http://169.254.169.254/ (cloud metadata) or an
RFC1918 host is still fetched — the guard's own docstring scoped this out.
Install a getaddrinfo guard in the download subprocess that re-validates every
resolved address at actual connect time, covering redirects and DNS rebinding
for any backend resolving through Python's socket module (urllib, requests).
Loopback is permitted so locally-configured proxies keep working; link-local,
RFC1918 and unique-local are blocked. Native resolvers (curl_cffi/libcurl via
--impersonate) bypass this and rely on network isolation as the backstop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-download chapter_template was validated only against literal ".."
in the template string, but yt-dlp expands %(section_title)s (and every
other field) from attacker-controlled metadata at download time. On POSIX
hosts yt-dlp does not neutralise a ".." path component, so a chapter titled
".." turns a guard-passing template like
"%(section_title)s/%(section_title)s/x.%(ext)s" into "../../x.mp4" and writes
outside DOWNLOAD_DIR. The same class of escape applies to any multi-segment
output template (default/playlist/channel) whose fields resolve to "..".
The template string can never see the "..": it only exists after expansion.
So move the check to the one point every output path flows through —
YoutubeDL.prepare_filename — via a _ConfinedYoutubeDL subclass that refuses
any resolved path outside the download/temp roots (fail closed). This covers
the main file, split-chapter files, thumbnails and subtitles in one place.
With the chokepoint authoritative, the scattered ingress string checks
(chapter_template, custom_name_prefix) and the weaker _output_dir_escapes
literal-prefix check are removed. Tests move from the ingress layer to the
chokepoint, exercising the real metadata-resolution vector.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds _AlbumArtistPostProcessor, a yt-dlp pre_process postprocessor that
fills album_artist from the '<artist> - Topic' channel/uploader signal,
falling back to the first credited artist, when album metadata exists
but no album artist is set.
Closes#1025.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
User-submitted URLs were passed straight to yt-dlp's generic extractor,
letting the server fetch internal endpoints (cloud metadata, loopback,
RFC1918 hosts). Add a url_guard.validate_url check at every URL ingress
(add, subscribe, and nested playlist recursion) that rejects non-http(s)
schemes and hosts resolving to non-global addresses, while leaving bare
video IDs and search prefixes untouched.
Sanitize path separators and .. segments in playlist/channel titles before they are baked into yt-dlp output templates, and refuse downloads whose resolved output directory escapes DOWNLOAD_DIR.
Addresses a full-project review. Backend correctness and availability:
- ytdl: cancel() only SIGKILLs the child's process group when the child
actually became its own group leader, so a race (or failed setpgrp)
can no longer kill the whole server; kill the group on cancel and on
shutdown to avoid orphaned ffmpeg children
- ytdl: dedicated ThreadPoolExecutor for download supervision so active
downloads can't starve extract_info / live probes on the default pool
- ytdl/main/subscriptions: route fire-and-forget tasks through a
bg_tasks helper that keeps a strong ref and logs failures
- subscriptions: run flat-playlist extraction in an executor and check
feeds with bounded concurrency so one slow feed can't block the loop;
set last_checked on failure so broken feeds aren't retried every 60s
- main: validate ids on /start & /delete and numeric env vars at startup;
return 400 (not 500) on bad subscriptions/update input; serve /history
from memory; move get_custom_dirs off the event loop; restrict t=
stripping to YouTube hosts; drop double percent-decode in state guard
- dl_formats/ytdl: enforce requested caption format via
FFmpegSubtitlesConvertor and strip VTT header metadata only in the
pre-cue region so real dialogue is preserved
- ytdl: throttle progress events, dedup adds against pending, clear
filename/size on error and reject out-of-dir trashcan deletes, pin
fork start-method on Linux only
Frontend:
- retry deletes the done record only after a successful re-add
- surface HTTP errors for delete/start and reset the deleting flag
- ignore late 'updated' events for rows no longer in the queue
- track table rows by map key; FileSizePipe uses base-1024
Also: HTTPS-aware Docker healthcheck, dead-code removal, and shared
helpers for path-containment and yt-dlp option merging. Adds/updates
unit tests throughout (250 backend tests passing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Detect YouTube channel tabs that yt-dlp reports as playlists so channel downloads use OUTPUT_TEMPLATE_CHANNEL and its empty fallback instead of OUTPUT_TEMPLATE_PLAYLIST.
The playlist/channel processing loop now sets playlist_count,
playlist_autonumber, n_entries, and __last_playlist_index on each
video entry so that templates like %(playlist_autonumber)s,
%(playlist_count)s, and %(playlist_index&{} - |)s resolve correctly
instead of showing NA.
Also updates _compact_persisted_entry to preserve n_entries and
__last_playlist_index across restarts.
Fixes#692
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/b5aeb55a-3197-4a14-b8b4-96c9a67796e8
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
Replace the hand-rolled _outtmpl_substitute_field() / _compile_outtmpl_pattern()
with a new _resolve_outtmpl_fields() that delegates to yt-dlp's
YoutubeDL.evaluate_outtmpl(). This gives playlist/channel output templates
access to yt-dlp's full template syntax: defaults (%(field|fallback)s),
conditional formatting (%(field&prefix {})s), math (%(field+N)d),
datetime formatting (%(field>%Y-%m-%d)s), and more.
Only field references whose root name matches the targeted prefix (e.g.
"playlist" or "channel") are resolved; all other references remain as
template placeholders for yt-dlp to fill during the actual download.
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/0ae5ff34-540f-4fc8-a81c-358fb92b7c15
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
Allow users to prefer a specific video codec (H.264, H.265, AV1, VP9)
when adding downloads. The selector filters available formats via
yt-dlp format strings, falling back to best available if the preferred
codec is not found. The completed downloads table now shows Quality
and Codec columns.
When adding a playlist, deleting a video from the queue causes it to be
re-added on the next loop iteration because queue.exists() returns False
for the deleted key. Track canceled URLs in a set so __add_entry skips
them.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Alex Shnitman