Commit Graph

246 Commits

Author SHA1 Message Date
Alex Shnitman fdfbfed5e2 fix: prevent playlist/channel title path traversal (closes GHSA-vh67-38x4-w8pc)
Sanitize path separators and .. segments in playlist/channel titles before they are baked into yt-dlp output templates, and refuse downloads whose resolved output directory escapes DOWNLOAD_DIR.
2026-07-13 23:07:39 +03:00
Alex Shnitman 3ea4732c5d fix: harden download lifecycle, subscriptions, and UI robustness
Addresses a full-project review. Backend correctness and availability:

- ytdl: cancel() only SIGKILLs the child's process group when the child
  actually became its own group leader, so a race (or failed setpgrp)
  can no longer kill the whole server; kill the group on cancel and on
  shutdown to avoid orphaned ffmpeg children
- ytdl: dedicated ThreadPoolExecutor for download supervision so active
  downloads can't starve extract_info / live probes on the default pool
- ytdl/main/subscriptions: route fire-and-forget tasks through a
  bg_tasks helper that keeps a strong ref and logs failures
- subscriptions: run flat-playlist extraction in an executor and check
  feeds with bounded concurrency so one slow feed can't block the loop;
  set last_checked on failure so broken feeds aren't retried every 60s
- main: validate ids on /start & /delete and numeric env vars at startup;
  return 400 (not 500) on bad subscriptions/update input; serve /history
  from memory; move get_custom_dirs off the event loop; restrict t=
  stripping to YouTube hosts; drop double percent-decode in state guard
- dl_formats/ytdl: enforce requested caption format via
  FFmpegSubtitlesConvertor and strip VTT header metadata only in the
  pre-cue region so real dialogue is preserved
- ytdl: throttle progress events, dedup adds against pending, clear
  filename/size on error and reject out-of-dir trashcan deletes, pin
  fork start-method on Linux only

Frontend:

- retry deletes the done record only after a successful re-add
- surface HTTP errors for delete/start and reset the deleting flag
- ignore late 'updated' events for rows no longer in the queue
- track table rows by map key; FileSizePipe uses base-1024

Also: HTTPS-aware Docker healthcheck, dead-code removal, and shared
helpers for path-containment and yt-dlp option merging. Adds/updates
unit tests throughout (250 backend tests passing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 08:08:14 +03:00
Alex Shnitman e2c777842e fix: honor OUTPUT_TEMPLATE for channel downloads (closes #1024)
Detect YouTube channel tabs that yt-dlp reports as playlists so channel downloads use OUTPUT_TEMPLATE_CHANNEL and its empty fallback instead of OUTPUT_TEMPLATE_PLAYLIST.
2026-07-10 09:49:59 +03:00
Markus Lanthaler ad90609c9b Don't mark a subscription as broken just because all entries are filtered out as they have already been downloaded
This happens if archive.txt is used
2026-07-08 22:54:33 +00:00
Matt Van Horn 54463baf0e fix: fsync parent dir after direct-write fallback for durability parity 2026-07-05 04:11:12 -07:00
Matt Van Horn b00d4785ee fix: serialize state before truncating in the direct-write fallback 2026-07-05 04:04:54 -07:00
Matt Van Horn 96e88a3555 fix: force 0600 on fallback state rewrites, not just creation 2026-07-05 04:00:59 -07:00
Matt Van Horn 49a46a7d1c fix: create fallback state file with owner-only 0600 permissions 2026-07-05 03:57:19 -07:00
Matt Van Horn 961b54aa83 fix: make fsync best-effort so only mkstemp/replace failures fall back 2026-07-05 03:53:22 -07:00
Matt Van Horn e0549d6c24 fix: surface real storage errors from direct-write fsync fallback 2026-07-05 03:49:15 -07:00
Matt Van Horn f315b75bb2 fix: limit atomic-write fallback to atomic-unsupported errnos 2026-07-05 03:44:57 -07:00
Matt Van Horn c2c129db61 fix: fall back to direct write when atomic state save hits EPERM on NFS 2026-07-05 03:42:07 -07:00
Alex Shnitman ce897ee009 fix open download of cookie files 2026-06-20 09:52:34 +03:00
Alex Shnitman 64d0d62878 fix empty PUBLIC_HOST_AUDIO_URL handling (closes #1010) 2026-06-16 21:47:07 +03:00
Alex Shnitman 5aa7d033e2 review fixes 2026-06-16 21:35:07 +03:00
Alex Shnitman 5429200fba support live streams (closes #302, closes #752, closes #978) 2026-06-13 17:39:14 +03:00
Alex Shnitman ee20512410 add option for following nightly yt-dlp releases (closes #999) 2026-06-06 09:42:26 +03:00
Alex Shnitman 5d96a581b9 allow filtering out members-only videos in subscriptions (closes #971) 2026-04-28 22:02:05 +03:00
Alex Shnitman 4f83174d05 implement time-clipped downloads (closes #969, replaces #907) 2026-04-26 23:07:50 +03:00
Alex Shnitman 91ee8312bf title filter for subscriptions (closes #968) 2026-04-26 22:51:48 +03:00
Alex Shnitman 23de9824f0 cr fixes 2026-04-21 16:13:58 +03:00
rdiaz738 0ea934c08f Updated import and fixed race condition 2026-04-20 17:24:16 -07:00
Alex Shnitman e9f979b349 fix yt-dlp options overrides (closes #958) 2026-04-18 08:46:29 +03:00
Alex Shnitman 1a32eba474 fix PUBLIC_HOST_URL without a trailing slash (closes #959) 2026-04-16 22:08:08 +03:00
Alex Shnitman 03f71fd257 fix asterisk CORS_ALLOWED_ORIGINS, mentioned in #955 2026-04-13 19:02:27 +03:00
Alex Shnitman 388aeb180d Merge branch 'bgervan/master' 2026-04-10 08:10:00 +03:00
Benjamin Gervan a6e8617ad8 Don't mark live streams as seen 2026-04-10 06:41:45 +02:00
az10b 0072d3488a Fix permissive CORS policy that allows cross-origin attacks
The on_prepare handler unconditionally reflected the Origin request
header into Access-Control-Allow-Origin, and Socket.IO was configured
with cors_allowed_origins='*'. This allowed any website to make
authenticated cross-origin requests to all API endpoints, enabling
cross-origin download initiation, cookie overwrite, and data deletion.

Replace the blanket origin reflection with an explicit allowlist via
the CORS_ALLOWED_ORIGINS environment variable. When unset, cross-origin
requests are denied by default. Users who need cross-origin access can
set CORS_ALLOWED_ORIGINS to a comma-separated list of trusted origins.
2026-04-09 19:45:51 -05:00
McSwindler d38d7bd1b1 fix: handle playlists that don't supply video ids 2026-04-09 10:15:11 -05:00
jacinli 373692ac65 fix: parse string boolean values when updating subscriptions 2026-04-05 14:05:59 +08:00
Alex Shnitman dd0f98d12f change option presets to be multi-select 2026-04-04 10:25:46 +03:00
copilot-swe-agent[bot] 6e9b2dd7b3 Gate manual yt-dlp overrides behind flag
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/31b4274d-cf48-4260-b73b-633cbcd2bb09

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-03 09:05:19 +00:00
copilot-swe-agent[bot] 565a715037 feat: add per-download yt-dlp presets and overrides
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/8a3119fc-63d1-4508-a196-8c50ff248812

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-03 06:16:12 +00:00
copilot-swe-agent[bot] d7eaaaa94b Add clarifying comments for n_entries and __last_playlist_index fields (closes #692)
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/b5aeb55a-3197-4a14-b8b4-96c9a67796e8

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-02 10:51:03 +03:00
copilot-swe-agent[bot] 981e6c1003 Propagate missing playlist context fields (playlist_count, playlist_autonumber, n_entries, __last_playlist_index)
The playlist/channel processing loop now sets playlist_count,
playlist_autonumber, n_entries, and __last_playlist_index on each
video entry so that templates like %(playlist_autonumber)s,
%(playlist_count)s, and %(playlist_index&{} - |)s resolve correctly
instead of showing NA.

Also updates _compact_persisted_entry to preserve n_entries and
__last_playlist_index across restarts.

Fixes #692

Agent-Logs-Url: https://github.com/alexta69/metube/sessions/b5aeb55a-3197-4a14-b8b4-96c9a67796e8

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-01 19:59:32 +00:00
copilot-swe-agent[bot] b17e1e5668 Add explanatory comment for fake STR_FORMAT_RE_TMPL key group in tests
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/0ae5ff34-540f-4fc8-a81c-358fb92b7c15

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-01 19:34:09 +00:00
copilot-swe-agent[bot] c1b5540332 Replace custom template substitution with yt-dlp's evaluate_outtmpl
Replace the hand-rolled _outtmpl_substitute_field() / _compile_outtmpl_pattern()
with a new _resolve_outtmpl_fields() that delegates to yt-dlp's
YoutubeDL.evaluate_outtmpl().  This gives playlist/channel output templates
access to yt-dlp's full template syntax: defaults (%(field|fallback)s),
conditional formatting (%(field&prefix {})s), math (%(field+N)d),
datetime formatting (%(field>%Y-%m-%d)s), and more.

Only field references whose root name matches the targeted prefix (e.g.
"playlist" or "channel") are resolved; all other references remain as
template placeholders for yt-dlp to fill during the actual download.

Agent-Logs-Url: https://github.com/alexta69/metube/sessions/0ae5ff34-540f-4fc8-a81c-358fb92b7c15

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-01 19:31:27 +00:00
Alex Shnitman 483575d24a add subscriptions; change persistence file format to JSON (closes #901, #76, #113, #170, #242, #444, #503, #555, #566) 2026-04-01 14:33:24 +03:00
Alex Shnitman 84c6418f91 fix pickle (closes #814) 2026-03-21 12:42:17 +02:00
Alex Shnitman a1f2fe3e73 implement tests 2026-03-20 13:12:31 +02:00
Alex Shnitman 7fa1fc7938 code review fixes 2026-03-15 20:53:13 +02:00
Alex Shnitman 5c321bfaca reoganize quality and codec selections 2026-03-13 19:47:36 +02:00
CyCl0ne 56826d33fd Add video codec selector and codec/quality columns in done list
Allow users to prefer a specific video codec (H.264, H.265, AV1, VP9)
when adding downloads. The selector filters available formats via
yt-dlp format strings, falling back to best available if the preferred
codec is not found. The completed downloads table now shows Quality
and Codec columns.
2026-03-09 08:59:01 +01:00
copilot-swe-agent[bot] 2736425e19 Revert #504 default change: restore original playlist/channel output templates
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-03-07 14:00:29 +00:00
copilot-swe-agent[bot] 0d905c0b61 Fix issues #898, #542, #561, #504
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-03-07 07:19:05 +00:00
copilot-swe-agent[bot] 6de4a56f28 Remove DEFAULT_DOWNLOAD_FOLDER feature
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-03-07 06:37:09 +00:00
copilot-swe-agent[bot] 1f4c4df847 Implement DEFAULT_DOWNLOAD_FOLDER and CLEAR_COMPLETED_AFTER features (#875, #869)
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-03-06 15:37:35 +00:00
Alex Shnitman 54e25484c5 some fixes in cookie upload functionality 2026-03-06 14:20:16 +02:00
ddmoney420 7cfb0c3a1d Add cookie file upload for authenticated downloads 2026-03-04 13:29:43 -07:00
ddmoney420 3587098e80 Fix deleted playlist videos being re-queued during add
When adding a playlist, deleting a video from the queue causes it to be
re-added on the next loop iteration because queue.exists() returns False
for the deleted key. Track canceled URLs in a set so __add_entry skips
them.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 16:17:50 -07:00