fix: harden download lifecycle, subscriptions, and UI robustness

Addresses a full-project review. Backend correctness and availability:

- ytdl: cancel() only SIGKILLs the child's process group when the child
  actually became its own group leader, so a race (or failed setpgrp)
  can no longer kill the whole server; kill the group on cancel and on
  shutdown to avoid orphaned ffmpeg children
- ytdl: dedicated ThreadPoolExecutor for download supervision so active
  downloads can't starve extract_info / live probes on the default pool
- ytdl/main/subscriptions: route fire-and-forget tasks through a
  bg_tasks helper that keeps a strong ref and logs failures
- subscriptions: run flat-playlist extraction in an executor and check
  feeds with bounded concurrency so one slow feed can't block the loop;
  set last_checked on failure so broken feeds aren't retried every 60s
- main: validate ids on /start & /delete and numeric env vars at startup;
  return 400 (not 500) on bad subscriptions/update input; serve /history
  from memory; move get_custom_dirs off the event loop; restrict t=
  stripping to YouTube hosts; drop double percent-decode in state guard
- dl_formats/ytdl: enforce requested caption format via
  FFmpegSubtitlesConvertor and strip VTT header metadata only in the
  pre-cue region so real dialogue is preserved
- ytdl: throttle progress events, dedup adds against pending, clear
  filename/size on error and reject out-of-dir trashcan deletes, pin
  fork start-method on Linux only

Frontend:

- retry deletes the done record only after a successful re-add
- surface HTTP errors for delete/start and reset the deleting flag
- ignore late 'updated' events for rows no longer in the queue
- track table rows by map key; FileSizePipe uses base-1024

Also: HTTPS-aware Docker healthcheck, dead-code removal, and shared
helpers for path-containment and yt-dlp option merging. Adds/updates
unit tests throughout (250 backend tests passing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Alex Shnitman
2026-07-12 08:08:14 +03:00
parent e2c777842e
commit 3ea4732c5d
21 changed files with 1392 additions and 166 deletions
+33 -1
View File
@@ -3,6 +3,21 @@ import copy
AUDIO_FORMATS = ("m4a", "mp3", "opus", "wav", "flac")
CAPTION_MODES = ("auto_only", "manual_only", "prefer_manual", "prefer_auto")
def merge_ytdl_option_layers(presets, overrides, presets_config) -> dict:
"""Overlay named presets (in order) then per-item overrides onto a fresh dict.
Does NOT include any base ``YTDL_OPTIONS`` — callers layer this on top of
their own base (a per-download build adds the global base; a subscription
scan relies on ``**config.YTDL_OPTIONS`` already being present in its
params). ``presets_config`` maps a preset name to its options dict.
"""
merged: dict = {}
for name in presets or []:
merged.update(presets_config.get(name, {}))
merged.update(overrides or {})
return merged
CODEC_FILTER_MAP = {
'h264': "[vcodec~='^(h264|avc)']",
'h265': "[vcodec~='^(h265|hevc)']",
@@ -43,6 +58,10 @@ def get_format(download_type: str, codec: str, format: str, quality: str) -> str
quality = (quality or "best").strip().lower()
if format.startswith("custom:"):
# Unreachable via the HTTP API (format is validated against a fixed
# set in main.py), but legacy persisted downloads may carry a
# custom: format from before that validation existed; removing this
# would crash PersistentQueue.load() for those records.
return format[7:]
if download_type == "thumbnail":
@@ -137,7 +156,20 @@ def get_opts(
requested_subtitle_format = (format or "srt").lower()
if requested_subtitle_format == "txt":
requested_subtitle_format = "srt"
opts["subtitlesformat"] = requested_subtitle_format
opts["subtitlesformat"] = f"{requested_subtitle_format}/best"
if requested_subtitle_format in ("srt", "vtt"):
# subtitlesformat above is only a preference: if the extractor
# doesn't natively offer this ext (e.g. YouTube has no native srt),
# yt-dlp silently falls back to whatever it has. ffmpeg can only
# convert to srt/vtt/ass/lrc, so only guarantee the requested
# container for those; other formats stay best-effort.
postprocessors.append(
{
"key": "FFmpegSubtitlesConvertor",
"format": requested_subtitle_format,
"when": "before_dl",
}
)
if mode == "manual_only":
opts["writesubtitles"] = True
opts["writeautomaticsub"] = False