yt-dlp reports a download 'finished' as soon as the media bytes have
landed, but the ffmpeg work that follows -- merging, re-encoding, chapter
splitting, sponsor removal -- routinely takes longer than the download
itself. The postprocessor hook only ever looked at MoveFiles and
SplitChapters finishing, so that whole phase said nothing: the row sat in
the Downloading table on a full, frozen progress bar, and the item counted
as neither active nor queued in the header stats.
Report a 'postprocessing' status when a postprocessor starts, and reuse
the indeterminate bar the UI already runs for 'preparing', labelled so a
long re-encode is distinguishable from a stall.
Measured on a real download re-encoded with the reporter's FFmpegCopyStream
config, the UI now receives:
[ 0.27s] downloading moving bar
[ 0.28s] finished <- yt-dlp, bytes are down
[ 0.28s] postprocessing animated "Post-processing"
[13.18s] finished done
i.e. 12.9s that used to render as a static 100% bar.
The hook is latched off once MoveFiles reports finished, since that branch
announces the finished file: a 'postprocessing' arriving afterwards would
flip the row back out of its completed state for no reason. It cannot fail
the download -- _download puts an unconditional 'finished' once download()
returns, so the terminal status is settled either way -- but the flicker
and the extra broadcast are both pointless. yt-dlp does run an 'after_move'
stage after MoveFiles, though every postprocessor MeTube configures is
'after_filter' or 'post_process', both of which precede it.
update_status drops a repeated 'postprocessing': yt-dlp's metaclass wraps
run() once per class in a postprocessor's MRO, so one whose subclass
overrides run reports started twice -- FFmpegCopyStream did exactly that in
the live run, three queued statuses collapsing to a single broadcast.
Extracted to _make_postprocessor_hook, mirroring _make_progress_hook, so
the ordering above is testable; every new regression test was confirmed to
fail with the fix backed out.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
yt-dlp documents 'filename' as always present in a progress hook, so the
update_status branch that stats it ran on every forwarded tick: throttled
to one every 0.5s per download, times MAX_CONCURRENT_DOWNLOADS. Those are
blocking syscalls on the event loop, and when the filesystem is slow each
one freezes every other request the server is serving -- which is what a
bare GET timing out at >10s looks like from outside.
The call was also useless while it was expensive. Until the download
finishes the bytes live in tmpfilename; 'filename' is the destination,
which does not exist yet, so os.path.exists() returned False and size
stayed None. It only yields a real value on a terminal status, and the
Downloading table has no size column, so nothing displayed it before
completion either way.
Stat only when the status is 'finished'. That covers both moments a file
genuinely exists at that path: yt-dlp's own finished status, and the
MoveFiles postprocessor reporting the final merged name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a "Remove sponsor segments" switch to the shared options panel,
building the same SponsorBlock + ModifyChapters pair the CLI's
--sponsorblock-remove sponsor does, and carries the flag through
subscriptions so the panel's control applies to both forms.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The image ships yt-dlp's bgutil PO token provider and starts it on
loopback, where the plugin dials it at http://127.0.0.1:4416. Since
482381d scoped the connect-time allowance to the configured proxy, the
download subprocess could no longer resolve it:
Refusing to connect to non-global address for host '127.0.0.1'
which surfaces as the plugin's "Error reaching GET .../ping". Metadata
extraction runs in the main process and installs no guard, so titles kept
resolving while the download itself ran without a token — and YouTube
increasingly answers those with 403.
The allowance already had the right shape for this; it was just named for
its only user. Endpoints the operator or the image configured are now
allowed as a class: install_socket_guard takes service_urls alongside
proxy_urls, and ytdl derives them from the bundled default plus any
base_url set through the youtubepot-bgutilhttp (or the deprecated youtube
getpot_bgutil_baseurl) extractor argument. The bundled server runs either
way, so it stays allowed when a base URL is configured.
Matching stays exact host:port on the configured string, so nothing else
on loopback opens up: a hostile media URL naming the endpoint reaches a
token server with two endpoints and nothing worth reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A "Remove sponsor segments" switch next to "Split by chapters" queues
the download with the same postprocessor pair the CLI's
--sponsorblock-remove sponsor builds (SponsorBlock + ModifyChapters).
The flag persists as a cookie like the other form options, survives in
the queue records, and is carried into retries.
The pair is registered above the chapter-splitting block: yt-dlp runs
same-stage postprocessors in list order, so ModifyChapters has to
rewrite the chapter list before FFmpegSplitChapters cuts the file up,
matching what the CLI builds for --sponsorblock-remove sponsor
--split-chapters. With both toggles on the other way around the chapter
files keep the sponsor segments and the removal desyncs the remaining
chapter timings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MeTube never sets ignoreerrors, so yt-dlp's trouble() raises DownloadError
instead of returning nonzero for failures like "No video formats found!".
That is the path issue #1047 reports, and it dropped the warnings that
explain the failure because only the nonzero-return branch attached them.
The exception branch now reports the retained warnings with the exception
text as the final line, so the actual error stays prominent under the
context. yt-dlp labels errors "ERROR:" but hands warnings to the logger
unlabelled, so a last warning that repeats the error text is skipped.
Retention is bounded to the last five distinct warnings: fragmented and
live downloads emit a warning per fragment, and the joined message is
persisted with the completed queue and broadcast to every client.
Co-authored-by: Matt Van Horn <mvanhorn@users.noreply.github.com>
Sites that put a description in the title produce output names past what
the filesystem accepts, and the download fails outright with
"[Errno 36] File name too long". The name is now shortened to fit inside
prepare_filename, which every output path already passes through, so the
main file, its chapter files, thumbnails and subtitles stay consistent.
The limit is read from the filesystem (PC_NAME_MAX, falling back to 255)
and counted in bytes, not characters: a title of accented or CJK
characters reaches it in a third of the characters. The extension is
kept, a cut landing inside a multi-byte character does not leave a broken
sequence, and a reserve is held back for the suffixes yt-dlp appends
afterwards -- '.part', '.ytdl', '.f<format_id>', '-Frag<n>' -- since it
is those that pushed the reported name over the limit.
Names that already fit are untouched, so nothing that downloads today
changes name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds __record_add_failure so a URL that fails before a real download starts
(unsupported/unextractable URL, SSRF-rejected, extraction error) appears in the
Completed list as a red-cross entry with retry and error-detail, instead of only
a transient toast and a server log line. Keyed by info.url like any errored
download. Includes _short_title_for_failed_url for a readable hostname title.
The frontend hunk removing the 'Click for details' hint from every error row was
dropped from this merge; that affordance (added in fd3aaea, #143) is kept.
Co-authored-by: streamer1122 <streamer1122@users.noreply.github.com>
Adds MusicMetadataPreProcessor (app/music_metadata.py), a pre_process
postprocessor that enriches the info dict using only extractor-owned fields:
track-number/total resolution, album-title fallback, and square-thumbnail
preference. Track numbers and album flow into files through yt-dlp's existing
FFmpegMetadata embedding, so no per-format tag-writing code and no new
dependency. The earlier mutagen writer and its download-failing
PostProcessingError path were dropped per review.
Co-authored-by: jahruz67 <jahruz67@users.noreply.github.com>
The per-download chapter_template was validated only against literal ".."
in the template string, but yt-dlp expands %(section_title)s (and every
other field) from attacker-controlled metadata at download time. On POSIX
hosts yt-dlp does not neutralise a ".." path component, so a chapter titled
".." turns a guard-passing template like
"%(section_title)s/%(section_title)s/x.%(ext)s" into "../../x.mp4" and writes
outside DOWNLOAD_DIR. The same class of escape applies to any multi-segment
output template (default/playlist/channel) whose fields resolve to "..".
The template string can never see the "..": it only exists after expansion.
So move the check to the one point every output path flows through —
YoutubeDL.prepare_filename — via a _ConfinedYoutubeDL subclass that refuses
any resolved path outside the download/temp roots (fail closed). This covers
the main file, split-chapter files, thumbnails and subtitles in one place.
With the chokepoint authoritative, the scattered ingress string checks
(chapter_template, custom_name_prefix) and the weaker _output_dir_escapes
literal-prefix check are removed. Tests move from the ingress layer to the
chokepoint, exercising the real metadata-resolution vector.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sanitize path separators and .. segments in playlist/channel titles before they are baked into yt-dlp output templates, and refuse downloads whose resolved output directory escapes DOWNLOAD_DIR.
Addresses a full-project review. Backend correctness and availability:
- ytdl: cancel() only SIGKILLs the child's process group when the child
actually became its own group leader, so a race (or failed setpgrp)
can no longer kill the whole server; kill the group on cancel and on
shutdown to avoid orphaned ffmpeg children
- ytdl: dedicated ThreadPoolExecutor for download supervision so active
downloads can't starve extract_info / live probes on the default pool
- ytdl/main/subscriptions: route fire-and-forget tasks through a
bg_tasks helper that keeps a strong ref and logs failures
- subscriptions: run flat-playlist extraction in an executor and check
feeds with bounded concurrency so one slow feed can't block the loop;
set last_checked on failure so broken feeds aren't retried every 60s
- main: validate ids on /start & /delete and numeric env vars at startup;
return 400 (not 500) on bad subscriptions/update input; serve /history
from memory; move get_custom_dirs off the event loop; restrict t=
stripping to YouTube hosts; drop double percent-decode in state guard
- dl_formats/ytdl: enforce requested caption format via
FFmpegSubtitlesConvertor and strip VTT header metadata only in the
pre-cue region so real dialogue is preserved
- ytdl: throttle progress events, dedup adds against pending, clear
filename/size on error and reject out-of-dir trashcan deletes, pin
fork start-method on Linux only
Frontend:
- retry deletes the done record only after a successful re-add
- surface HTTP errors for delete/start and reset the deleting flag
- ignore late 'updated' events for rows no longer in the queue
- track table rows by map key; FileSizePipe uses base-1024
Also: HTTPS-aware Docker healthcheck, dead-code removal, and shared
helpers for path-containment and yt-dlp option merging. Adds/updates
unit tests throughout (250 backend tests passing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The playlist/channel processing loop now sets playlist_count,
playlist_autonumber, n_entries, and __last_playlist_index on each
video entry so that templates like %(playlist_autonumber)s,
%(playlist_count)s, and %(playlist_index&{} - |)s resolve correctly
instead of showing NA.
Also updates _compact_persisted_entry to preserve n_entries and
__last_playlist_index across restarts.
Fixes#692
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/b5aeb55a-3197-4a14-b8b4-96c9a67796e8
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
Replace the hand-rolled _outtmpl_substitute_field() / _compile_outtmpl_pattern()
with a new _resolve_outtmpl_fields() that delegates to yt-dlp's
YoutubeDL.evaluate_outtmpl(). This gives playlist/channel output templates
access to yt-dlp's full template syntax: defaults (%(field|fallback)s),
conditional formatting (%(field&prefix {})s), math (%(field+N)d),
datetime formatting (%(field>%Y-%m-%d)s), and more.
Only field references whose root name matches the targeted prefix (e.g.
"playlist" or "channel") are resolved; all other references remain as
template placeholders for yt-dlp to fill during the actual download.
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/0ae5ff34-540f-4fc8-a81c-358fb92b7c15
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
Alex Shnitman