Commit Graph

777 Commits

Author SHA1 Message Date
Alex Shnitman 1b02a99510 fix: re-validate outbound connections at fetch time against internal hosts
validate_url only inspects the submitted URL string. yt-dlp then follows HTTP
redirects and resolves media URLs from remote metadata without re-checking, so
an allowed URL that 302s to http://169.254.169.254/ (cloud metadata) or an
RFC1918 host is still fetched — the guard's own docstring scoped this out.

Install a getaddrinfo guard in the download subprocess that re-validates every
resolved address at actual connect time, covering redirects and DNS rebinding
for any backend resolving through Python's socket module (urllib, requests).
Loopback is permitted so locally-configured proxies keep working; link-local,
RFC1918 and unique-local are blocked. Native resolvers (curl_cffi/libcurl via
--impersonate) bypass this and rely on network isolation as the backstop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 06:55:29 +03:00
Alex Shnitman ebcfe577bc fix: fail closed when an SSRF-guarded host cannot be resolved
validate_url() previously returned None (allow) on socket.gaierror, so a
host that failed to resolve at check time was passed straight to yt-dlp. A
host we cannot resolve is a host we cannot verify as non-internal, and it may
resolve differently when yt-dlp fetches it. Reject it instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 23:45:08 +03:00
Alex Shnitman 3bd2c3e366 fix: enforce download-dir containment at the resolved-path chokepoint
The per-download chapter_template was validated only against literal ".."
in the template string, but yt-dlp expands %(section_title)s (and every
other field) from attacker-controlled metadata at download time. On POSIX
hosts yt-dlp does not neutralise a ".." path component, so a chapter titled
".." turns a guard-passing template like
"%(section_title)s/%(section_title)s/x.%(ext)s" into "../../x.mp4" and writes
outside DOWNLOAD_DIR. The same class of escape applies to any multi-segment
output template (default/playlist/channel) whose fields resolve to "..".

The template string can never see the "..": it only exists after expansion.
So move the check to the one point every output path flows through —
YoutubeDL.prepare_filename — via a _ConfinedYoutubeDL subclass that refuses
any resolved path outside the download/temp roots (fail closed). This covers
the main file, split-chapter files, thumbnails and subtitles in one place.

With the chokepoint authoritative, the scattered ingress string checks
(chapter_template, custom_name_prefix) and the weaker _output_dir_escapes
literal-prefix check are removed. Tests move from the ingress layer to the
chokepoint, exercising the real metadata-resolution vector.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 23:44:43 +03:00
Alex Shnitman 707f700609 ci: update releases in place instead of delete-and-recreate
The same-day rerun path deleted and recreated the release tag within
seconds, which corrupted GitHub's release index (release 2026.07.05 was
hidden from the public list and floated to the top for maintainers).
Replace it with gh release create/edit: force-move the tag in place on
reruns so it matches the rebuilt Docker image, exclude today's tag when
collecting release notes so reruns cover the whole day, and pin the
created tag to the triggering commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 16:44:52 +03:00
Alex Shnitman c519f45908 chore: rework issue and discussion templates around scope policy
Issue forms: trim prerequisite checkboxes to the two that matter, make the
yt-dlp test field explicitly conditional (UI bugs write "UI bug"), lead the
feature form with the scope line and the already-decided list, and retire
the question template in favor of Discussions Q&A.

Discussion forms: rename q-and-a.yml to q-a.yml so it matches the actual
category slug (it never applied), add ideas.yml for the category where
feature requests actually land, drop configuration-help.yml (no matching
category) and the inert config.yml (blank_discussions_enabled is not a
GitHub feature), and remove checkboxes from low-stakes discussion forms.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 13:50:34 +03:00
Alex Shnitman a23d1689e3 docs: add SECURITY.md (private vulnerability reporting)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026.07.18
2026-07-18 09:24:04 +03:00
Alex Shnitman 4b05022b91 feat: graceful cancel — SIGINT with SIGKILL escalation so partial files are finalized (closes #438)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 09:23:48 +03:00
Alex Shnitman 0dc9b0b3d6 ci: enforce the 25k Docker Hub limit on README.md
Runs only when README.md changes (which the build workflow ignores),
so the limit is checked exactly when it can be exceeded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:39:31 +03:00
Alex Shnitman 96f5ffe34a docs: restructure README around the wiki as companion documentation
- Move bookmarklet code and Apache/Caddy/swag reverse-proxy configs to
  the wiki; link them
- Group browser extensions, bookmarklets, iOS Shortcut, and Raycast
  under one 'Sending links to MeTube' section with shared CORS/HTTPS
  prerequisites stated once
- Move Runtime & Permissions (PUID/PGID/UMASK) to the top of the
  env-var reference; slim the CORS_ALLOWED_ORIGINS entry
- Link the new Subscriptions guide and Troubleshooting FAQ wiki pages
- State the project scope line in 'Submitting feature requests'
- Docker Compose naming, multi-arch note, trailing whitespace

24,830 -> 21,571 chars against the 25k Docker Hub limit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:39:20 +03:00
Alex Shnitman 1eafecd1cc docs: add test gotchas, option checklist, and security invariants to AGENTS.md
- Test-running traps: repo-root cwd requirement, frontend-build ordering,
  the yt_dlp stub quirk in test_ytdl_utils.py
- Note that master is continuously released
- Checklist for adding a per-download option (the split_by_chapters
  pattern, including the commonly missed pieces)
- Security invariants: SSRF guard for URLs, path helpers for anything
  derived from untrusted metadata
- Conventions: OnPush/markForCheck, compact persisted state, yt-dlp
  postprocessor list ordering

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:14:33 +03:00
Alex Shnitman 8fe81dea18 docs: encode project scope boundary in AGENTS.md
Document the line between in-scope work (improving the download-time
write, surfacing yt-dlp built-ins) and out-of-scope work (post-download
tag editing, external metadata lookups, library organization), so
agent-assisted contributions can check their plans against it before
writing code. Follows the decisions on #1025, #1026/#1027, #1028, #1031.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:08:33 +03:00
Alex f054d84108 Merge pull request #1029 from tjelite1986/readme-library-pairing
docs: document pairing MeTube with a music tagger
2026-07-17 15:00:51 +03:00
tjelite1986 fa02717e12 docs: document pairing MeTube with a music tagger
Short section pointing beets / Picard / Lidarr at AUDIO_DOWNLOAD_DIR,
as suggested in #1027.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 22:39:24 +02:00
Alex Shnitman 9ca78be199 Merge PR #1025: fill missing album-artist metadata for audio downloads
Adds _AlbumArtistPostProcessor, a yt-dlp pre_process postprocessor that
fills album_artist from the '<artist> - Topic' channel/uploader signal,
falling back to the first credited artist, when album metadata exists
but no album artist is set.

Closes #1025.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026.07.16
2026-07-16 23:03:17 +03:00
Alex Shnitman 8071611a84 upgrade dependencies 2026-07-16 23:01:23 +03:00
Your GitHub Name 220f991fae fix: prefer topic channel for album artist 2026-07-16 12:34:03 -07:00
Alex Shnitman 6d0528783c fix: block SSRF via user-submitted URLs
User-submitted URLs were passed straight to yt-dlp's generic extractor,
letting the server fetch internal endpoints (cloud metadata, loopback,
RFC1918 hosts). Add a url_guard.validate_url check at every URL ingress
(add, subscribe, and nested playlist recursion) that rejects non-http(s)
schemes and hosts resolving to non-global addresses, while leaving bare
video IDs and search prefixes untouched.
2026-07-16 21:31:51 +03:00
Your GitHub Name c104e30451 feat: add AlbumArtistPostProcessor to fill missing album-artist metadata 2026-07-15 15:40:36 -07:00
Alex Shnitman fdfbfed5e2 fix: prevent playlist/channel title path traversal (closes GHSA-vh67-38x4-w8pc)
Sanitize path separators and .. segments in playlist/channel titles before they are baked into yt-dlp output templates, and refuse downloads whose resolved output directory escapes DOWNLOAD_DIR.
2026.07.13
2026-07-13 23:07:39 +03:00
Alex Shnitman 3ea4732c5d fix: harden download lifecycle, subscriptions, and UI robustness
Addresses a full-project review. Backend correctness and availability:

- ytdl: cancel() only SIGKILLs the child's process group when the child
  actually became its own group leader, so a race (or failed setpgrp)
  can no longer kill the whole server; kill the group on cancel and on
  shutdown to avoid orphaned ffmpeg children
- ytdl: dedicated ThreadPoolExecutor for download supervision so active
  downloads can't starve extract_info / live probes on the default pool
- ytdl/main/subscriptions: route fire-and-forget tasks through a
  bg_tasks helper that keeps a strong ref and logs failures
- subscriptions: run flat-playlist extraction in an executor and check
  feeds with bounded concurrency so one slow feed can't block the loop;
  set last_checked on failure so broken feeds aren't retried every 60s
- main: validate ids on /start & /delete and numeric env vars at startup;
  return 400 (not 500) on bad subscriptions/update input; serve /history
  from memory; move get_custom_dirs off the event loop; restrict t=
  stripping to YouTube hosts; drop double percent-decode in state guard
- dl_formats/ytdl: enforce requested caption format via
  FFmpegSubtitlesConvertor and strip VTT header metadata only in the
  pre-cue region so real dialogue is preserved
- ytdl: throttle progress events, dedup adds against pending, clear
  filename/size on error and reject out-of-dir trashcan deletes, pin
  fork start-method on Linux only

Frontend:

- retry deletes the done record only after a successful re-add
- surface HTTP errors for delete/start and reset the deleting flag
- ignore late 'updated' events for rows no longer in the queue
- track table rows by map key; FileSizePipe uses base-1024

Also: HTTPS-aware Docker healthcheck, dead-code removal, and shared
helpers for path-containment and yt-dlp option merging. Adds/updates
unit tests throughout (250 backend tests passing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026.07.12
2026-07-12 08:08:14 +03:00
Alex Shnitman e2c777842e fix: honor OUTPUT_TEMPLATE for channel downloads (closes #1024)
Detect YouTube channel tabs that yt-dlp reports as playlists so channel downloads use OUTPUT_TEMPLATE_CHANNEL and its empty fallback instead of OUTPUT_TEMPLATE_PLAYLIST.
2026.07.10
2026-07-10 09:49:59 +03:00
Alex Shnitman c34a18de7a upgrade dependencies 2026-07-10 09:48:48 +03:00
Alex d6bcf182c5 Merge pull request #1023 from lanthaler/no-entries-archive
Don't mark a subscription as broken just because all entries are filtered out as they have already been downloaded
2026.07.09
2026-07-09 08:17:00 +03:00
Markus Lanthaler ad90609c9b Don't mark a subscription as broken just because all entries are filtered out as they have already been downloaded
This happens if archive.txt is used
2026-07-08 22:54:33 +00:00
Alex 5315630ab0 Merge pull request #1021 from mvanhorn/fix/960-atomic-store-nfs-eperm
fix: fall back to a direct write when AtomicJsonStore.save cannot use a temp file (NFS EPERM)
2026.07.05
2026-07-05 21:26:49 +03:00
Matt Van Horn 54463baf0e fix: fsync parent dir after direct-write fallback for durability parity 2026-07-05 04:11:12 -07:00
Matt Van Horn b00d4785ee fix: serialize state before truncating in the direct-write fallback 2026-07-05 04:04:54 -07:00
Matt Van Horn 96e88a3555 fix: force 0600 on fallback state rewrites, not just creation 2026-07-05 04:00:59 -07:00
Matt Van Horn 49a46a7d1c fix: create fallback state file with owner-only 0600 permissions 2026-07-05 03:57:19 -07:00
Matt Van Horn 961b54aa83 fix: make fsync best-effort so only mkstemp/replace failures fall back 2026-07-05 03:53:22 -07:00
Matt Van Horn e0549d6c24 fix: surface real storage errors from direct-write fsync fallback 2026-07-05 03:49:15 -07:00
Matt Van Horn f315b75bb2 fix: limit atomic-write fallback to atomic-unsupported errnos 2026-07-05 03:44:57 -07:00
Matt Van Horn c2c129db61 fix: fall back to direct write when atomic state save hits EPERM on NFS 2026-07-05 03:42:07 -07:00
Alex 363f159a0a Merge pull request #1020 from alexta69/copilot/fix-dockerhub-build-push
Pin pnpm version to fix Docker build failure
2026-07-05 07:51:08 +03:00
copilot-swe-agent[bot] 38c0ca22f4 Pin pnpm version in packageManager field to fix Docker build
corepack prepare pnpm --activate without a version was resolving to
12.0.0-alpha.0 (broken pre-release), causing the dockerhub-build-push
job to fail. Adding packageManager field pins it to a stable release.
2026-07-05 04:21:32 +00:00
copilot-swe-agent[bot] 24ae8f0742 Initial plan 2026-07-05 04:18:28 +00:00
AutoUpdater 0a946cc352 upgrade yt-dlp from 2026.6.9 to 2026.7.4 2026-07-05 00:28:09 +00:00
Alex Shnitman 51fd203b71 upgrade to Angular 22 2026.06.28 2026-06-28 21:12:20 +03:00
Alex Shnitman d136344c26 upgrade dependencies 2026-06-28 21:08:08 +03:00
dependabot[bot] 33f1412fac Bump actions/checkout from 6 to 7 in the github-actions group
Bumps the github-actions group with 1 update: [actions/checkout](https://github.com/actions/checkout).


Updates `actions/checkout` from 6 to 7
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-06-28 21:06:58 +03:00
Alex Shnitman ce897ee009 fix open download of cookie files 2026.06.20 2026-06-20 09:52:34 +03:00
Alex Shnitman dd1b4c2436 upgrade dependencies 2026-06-20 09:52:34 +03:00
Alex 8752b500d6 Merge pull request #1004 from akeeton/docker-audio-download-dir
Create AUDIO_DOWNLOAD_DIR in Docker image
2026.06.18
2026-06-18 06:45:50 +03:00
Andrew Keeton 04b9366764 Incorporate PR feedback
Move the default assignment of AUDIO_DOWNLOAD_DIR from the Dockerfile to docker-entrypoint.sh, and change the default value from "/downloads" to $DOWNLOAD_DIR.
2026-06-17 17:17:44 -04:00
Alex Shnitman b73e95f405 upgrade dependencies 2026.06.16 2026-06-16 21:57:07 +03:00
Alex Shnitman 64d0d62878 fix empty PUBLIC_HOST_AUDIO_URL handling (closes #1010) 2026-06-16 21:47:07 +03:00
Alex Shnitman 37f7af0555 fix batch download (closes #1008) 2026-06-16 21:37:05 +03:00
Alex Shnitman 5aa7d033e2 review fixes 2026-06-16 21:35:07 +03:00
Alex Shnitman 5429200fba support live streams (closes #302, closes #752, closes #978) 2026.06.13 2026-06-13 17:39:14 +03:00
Alex Shnitman 72d60ea55a upgrade dependencies 2026-06-12 12:45:38 +03:00