Compare commits

...

21 Commits

Author SHA1 Message Date
Alex Shnitman e061a8a5ba feat: ALLOW_PRIVATE_ADDRESSES to opt out of the SSRF checks (closes #1036)
The SSRF guard rejects any address that isn't globally routable, which breaks
proxy/VPN setups that resolve hosts into private or special-use ranges. The
reported case is Fake-IP clients (sing-box, Clash, Mihomo) that map YouTube to
the RFC 2544 benchmarking range 198.18.0.0/15 to tunnel the traffic; MeTube
rejected it with "Refusing to fetch internal address" even though yt-dlp itself
handles it fine.

Add a boolean ALLOW_PRIVATE_ADDRESSES (default false) that, when set, skips both
layers: validate_url returns after scheme validation without the internal-host
checks, and the connect-time socket guard is not installed. Threaded to the
download subprocess via Download so the guard sees it too. Scheme validation
(http/https only) still applies. Documented in the README as a trust-your-network
opt-out that disables SSRF protection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 07:44:38 +03:00
Alex Shnitman 13cb65d931 docs: document the SSRF guard's connect-time coverage limitations
The connect-time getaddrinfo guard added for redirect/rebinding SSRF covers
only the download subprocess: metadata extraction runs in the main process
(where a process-wide guard would reject the server's own HOST=0.0.0.0 bind),
and native curl_cffi/libcurl resolution used by --impersonate bypasses Python's
socket module. Record both in url_guard's docstring and at the extraction site
so the boundary is explicit; network isolation remains the backstop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 07:41:01 +03:00
Alex Shnitman 1b02a99510 fix: re-validate outbound connections at fetch time against internal hosts
validate_url only inspects the submitted URL string. yt-dlp then follows HTTP
redirects and resolves media URLs from remote metadata without re-checking, so
an allowed URL that 302s to http://169.254.169.254/ (cloud metadata) or an
RFC1918 host is still fetched — the guard's own docstring scoped this out.

Install a getaddrinfo guard in the download subprocess that re-validates every
resolved address at actual connect time, covering redirects and DNS rebinding
for any backend resolving through Python's socket module (urllib, requests).
Loopback is permitted so locally-configured proxies keep working; link-local,
RFC1918 and unique-local are blocked. Native resolvers (curl_cffi/libcurl via
--impersonate) bypass this and rely on network isolation as the backstop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 06:55:29 +03:00
Alex Shnitman ebcfe577bc fix: fail closed when an SSRF-guarded host cannot be resolved
validate_url() previously returned None (allow) on socket.gaierror, so a
host that failed to resolve at check time was passed straight to yt-dlp. A
host we cannot resolve is a host we cannot verify as non-internal, and it may
resolve differently when yt-dlp fetches it. Reject it instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 23:45:08 +03:00
Alex Shnitman 3bd2c3e366 fix: enforce download-dir containment at the resolved-path chokepoint
The per-download chapter_template was validated only against literal ".."
in the template string, but yt-dlp expands %(section_title)s (and every
other field) from attacker-controlled metadata at download time. On POSIX
hosts yt-dlp does not neutralise a ".." path component, so a chapter titled
".." turns a guard-passing template like
"%(section_title)s/%(section_title)s/x.%(ext)s" into "../../x.mp4" and writes
outside DOWNLOAD_DIR. The same class of escape applies to any multi-segment
output template (default/playlist/channel) whose fields resolve to "..".

The template string can never see the "..": it only exists after expansion.
So move the check to the one point every output path flows through —
YoutubeDL.prepare_filename — via a _ConfinedYoutubeDL subclass that refuses
any resolved path outside the download/temp roots (fail closed). This covers
the main file, split-chapter files, thumbnails and subtitles in one place.

With the chokepoint authoritative, the scattered ingress string checks
(chapter_template, custom_name_prefix) and the weaker _output_dir_escapes
literal-prefix check are removed. Tests move from the ingress layer to the
chokepoint, exercising the real metadata-resolution vector.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 23:44:43 +03:00
Alex Shnitman 707f700609 ci: update releases in place instead of delete-and-recreate
The same-day rerun path deleted and recreated the release tag within
seconds, which corrupted GitHub's release index (release 2026.07.05 was
hidden from the public list and floated to the top for maintainers).
Replace it with gh release create/edit: force-move the tag in place on
reruns so it matches the rebuilt Docker image, exclude today's tag when
collecting release notes so reruns cover the whole day, and pin the
created tag to the triggering commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 16:44:52 +03:00
Alex Shnitman c519f45908 chore: rework issue and discussion templates around scope policy
Issue forms: trim prerequisite checkboxes to the two that matter, make the
yt-dlp test field explicitly conditional (UI bugs write "UI bug"), lead the
feature form with the scope line and the already-decided list, and retire
the question template in favor of Discussions Q&A.

Discussion forms: rename q-and-a.yml to q-a.yml so it matches the actual
category slug (it never applied), add ideas.yml for the category where
feature requests actually land, drop configuration-help.yml (no matching
category) and the inert config.yml (blank_discussions_enabled is not a
GitHub feature), and remove checkboxes from low-stakes discussion forms.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 13:50:34 +03:00
Alex Shnitman a23d1689e3 docs: add SECURITY.md (private vulnerability reporting)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 09:24:04 +03:00
Alex Shnitman 4b05022b91 feat: graceful cancel — SIGINT with SIGKILL escalation so partial files are finalized (closes #438)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 09:23:48 +03:00
Alex Shnitman 0dc9b0b3d6 ci: enforce the 25k Docker Hub limit on README.md
Runs only when README.md changes (which the build workflow ignores),
so the limit is checked exactly when it can be exceeded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:39:31 +03:00
Alex Shnitman 96f5ffe34a docs: restructure README around the wiki as companion documentation
- Move bookmarklet code and Apache/Caddy/swag reverse-proxy configs to
  the wiki; link them
- Group browser extensions, bookmarklets, iOS Shortcut, and Raycast
  under one 'Sending links to MeTube' section with shared CORS/HTTPS
  prerequisites stated once
- Move Runtime & Permissions (PUID/PGID/UMASK) to the top of the
  env-var reference; slim the CORS_ALLOWED_ORIGINS entry
- Link the new Subscriptions guide and Troubleshooting FAQ wiki pages
- State the project scope line in 'Submitting feature requests'
- Docker Compose naming, multi-arch note, trailing whitespace

24,830 -> 21,571 chars against the 25k Docker Hub limit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:39:20 +03:00
Alex Shnitman 1eafecd1cc docs: add test gotchas, option checklist, and security invariants to AGENTS.md
- Test-running traps: repo-root cwd requirement, frontend-build ordering,
  the yt_dlp stub quirk in test_ytdl_utils.py
- Note that master is continuously released
- Checklist for adding a per-download option (the split_by_chapters
  pattern, including the commonly missed pieces)
- Security invariants: SSRF guard for URLs, path helpers for anything
  derived from untrusted metadata
- Conventions: OnPush/markForCheck, compact persisted state, yt-dlp
  postprocessor list ordering

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:14:33 +03:00
Alex Shnitman 8fe81dea18 docs: encode project scope boundary in AGENTS.md
Document the line between in-scope work (improving the download-time
write, surfacing yt-dlp built-ins) and out-of-scope work (post-download
tag editing, external metadata lookups, library organization), so
agent-assisted contributions can check their plans against it before
writing code. Follows the decisions on #1025, #1026/#1027, #1028, #1031.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:08:33 +03:00
Alex f054d84108 Merge pull request #1029 from tjelite1986/readme-library-pairing
docs: document pairing MeTube with a music tagger
2026-07-17 15:00:51 +03:00
tjelite1986 fa02717e12 docs: document pairing MeTube with a music tagger
Short section pointing beets / Picard / Lidarr at AUDIO_DOWNLOAD_DIR,
as suggested in #1027.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 22:39:24 +02:00
Alex Shnitman 9ca78be199 Merge PR #1025: fill missing album-artist metadata for audio downloads
Adds _AlbumArtistPostProcessor, a yt-dlp pre_process postprocessor that
fills album_artist from the '<artist> - Topic' channel/uploader signal,
falling back to the first credited artist, when album metadata exists
but no album artist is set.

Closes #1025.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 23:03:17 +03:00
Alex Shnitman 8071611a84 upgrade dependencies 2026-07-16 23:01:23 +03:00
Your GitHub Name 220f991fae fix: prefer topic channel for album artist 2026-07-16 12:34:03 -07:00
Alex Shnitman 6d0528783c fix: block SSRF via user-submitted URLs
User-submitted URLs were passed straight to yt-dlp's generic extractor,
letting the server fetch internal endpoints (cloud metadata, loopback,
RFC1918 hosts). Add a url_guard.validate_url check at every URL ingress
(add, subscribe, and nested playlist recursion) that rejects non-http(s)
schemes and hosts resolving to non-global addresses, while leaving bare
video IDs and search prefixes untouched.
2026-07-16 21:31:51 +03:00
Your GitHub Name c104e30451 feat: add AlbumArtistPostProcessor to fill missing album-artist metadata 2026-07-15 15:40:36 -07:00
Alex Shnitman fdfbfed5e2 fix: prevent playlist/channel title path traversal (closes GHSA-vh67-38x4-w8pc)
Sanitize path separators and .. segments in playlist/channel titles before they are baked into yt-dlp output templates, and refuse downloads whose resolved output directory escapes DOWNLOAD_DIR.
2026-07-13 23:07:39 +03:00
25 changed files with 1444 additions and 749 deletions
-1
View File
@@ -1 +0,0 @@
blank_discussions_enabled: false
@@ -1,65 +0,0 @@
name: ⚙️ Configuration Help
description: Get help with MeTube configuration and setup
title: "[Config]: "
labels: ["configuration", "help"]
assignees: []
body:
- type: checkboxes
id: prerequisites
attributes:
label: Prerequisites
description: Please confirm you have completed these steps before asking for configuration help
options:
- label: I have searched existing discussions and issues for similar configuration problems
required: true
- label: I have read the [configuration section](https://github.com/alexta69/metube#%EF%B8%8F-configuration-via-environment-variables) in the README
required: true
- label: I have checked the [Wiki](https://github.com/alexta69/metube/wiki) for configuration examples
required: true
- type: markdown
attributes:
value: |
## Configuration Resources
Before asking for help, please check these resources:
- **[Configuration Guide](https://github.com/alexta69/metube#%EF%B8%8F-configuration-via-environment-variables)** - All available environment variables
- **[YTDL_OPTIONS Cookbook](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook)** - Common yt-dlp configurations
- **[OUTPUT_TEMPLATE Cookbook](https://github.com/alexta69/metube/wiki/OUTPUT_TEMPLATE-Cookbook)** - Filename template examples
- type: textarea
id: configuration-goal
attributes:
label: What are you trying to configure?
description: Describe what you want to achieve with your MeTube configuration
placeholder: |
What specific behavior are you trying to achieve?
What's not working as expected?
What have you tried so far?
validations:
required: true
- type: textarea
id: additional-context
attributes:
label: Additional Context
description: Your configuration, environment details, errors, and any other helpful information
placeholder: |
Current Configuration:
```yaml
# Your docker-compose.yml or environment variables
```
Environment Details:
- MeTube version: [e.g., latest, specific version]
- Docker image: [e.g., ghcr.io/alexta69/metube:latest]
- Operating System: [e.g., Ubuntu 20.04, Windows 10, macOS 12]
Error Messages or Issues:
[Paste any error messages, logs, or unexpected behavior here]
Other relevant information:
[Screenshots, examples, etc.]
validations:
required: true
+4 -30
View File
@@ -1,30 +1,13 @@
name: 💬 General Discussion
description: Start a general discussion about MeTube
title: "[Discussion]: "
labels: ["discussion"]
assignees: []
body:
- type: checkboxes
id: prerequisites
attributes:
label: Prerequisites
description: Please confirm you have completed these steps before starting a discussion
options:
- label: I have searched existing discussions to ensure this topic hasn't been discussed before
required: true
- label: I have read the [README](https://github.com/alexta69/metube#readme) and relevant sections
required: true
- type: markdown
attributes:
value: |
## Discussion Guidelines
This is for general discussions about MeTube. For specific issues, please use:
- **Bug reports** → Use the Bug Report issue template
- **Feature requests** → Use the Feature Request issue template
- **Questions** → Use the Question issue template
This is for general discussions about MeTube. For specific topics, better homes exist:
- **Bug reports** → [open an issue](https://github.com/alexta69/metube/issues/new?template=bug_report.yml)
- **Feature requests** → [open an issue](https://github.com/alexta69/metube/issues/new?template=feature_request.yml) or post in [Ideas](https://github.com/alexta69/metube/discussions/categories/ideas)
- **Questions** → post in [Q&A](https://github.com/alexta69/metube/discussions/categories/q-a)
- type: textarea
id: discussion-topic
@@ -34,12 +17,3 @@ body:
placeholder: Please provide a clear topic for discussion
validations:
required: true
- type: textarea
id: additional-context
attributes:
label: Additional Context
description: Any other information that might be helpful for the discussion
placeholder: Links, examples, or other relevant information
validations:
required: false
+31
View File
@@ -0,0 +1,31 @@
body:
- type: markdown
attributes:
value: |
## Read this first — MeTube's scope
MeTube's scope is deliberately narrow: **it downloads well and stops once the file is written.**
Ideas that improve the download itself are welcome — usually as community PRs, since the
maintainer is unlikely to implement requests ([details](https://github.com/alexta69/metube#-submitting-feature-requests)).
Post-download file management is out of scope regardless of implementation quality.
**Already decided — please don't re-request:**
- 🔒 Built-in authentication / password → use a reverse proxy ([#931](https://github.com/alexta69/metube/issues/931), [wiki](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations))
- 🔔 Notifications on download completion → available today via the `Exec` postprocessor ([recipe](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook))
- ✏️ Renaming, converting, tagging, or organizing files after download → out of scope ([#495](https://github.com/alexta69/metube/issues/495), [#1027](https://github.com/alexta69/metube/issues/1027))
- 🎚️ Pre-download format/quality/audio-track picker → tracked in [#1032](https://github.com/alexta69/metube/issues/1032)
- 📋 Pre-download playlist item browser → tracked in [#1030](https://github.com/alexta69/metube/issues/1030)
- 🎛️ Config switches to disable individual features → declined ([#976](https://github.com/alexta69/metube/issues/976))
- ⏰ Download scheduler → declined; add items with auto-start off and start them when you like ([#838](https://github.com/alexta69/metube/issues/838))
- type: textarea
id: idea
attributes:
label: Your Idea
description: Describe the idea and what problem it solves
placeholder: |
What would you like to see?
What problem does it solve?
Would you be willing to implement it as a PR?
validations:
required: true
+23
View File
@@ -0,0 +1,23 @@
body:
- type: markdown
attributes:
value: |
## Quick Resources
Your question may already be answered here:
- **[README](https://github.com/alexta69/metube#readme)** — complete setup and configuration guide
- **[Troubleshooting FAQ](https://github.com/alexta69/metube/wiki/Troubleshooting-FAQ)** — common problems and their fixes
- **[YTDL_OPTIONS Cookbook](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook)** — common yt-dlp configurations (notifications, metadata, audio extraction, ...)
- **[OUTPUT_TEMPLATE Cookbook](https://github.com/alexta69/metube/wiki/OUTPUT_TEMPLATE-Cookbook)** — filename template examples
- type: textarea
id: question
attributes:
label: Your Question
description: What would you like to know about MeTube?
placeholder: |
What are you trying to achieve?
What's your current setup (docker-compose / environment variables)?
What have you tried so far, and what happened (errors, logs)?
validations:
required: true
-60
View File
@@ -1,60 +0,0 @@
name: ❓ Q&A
description: Ask a question and get answers from the community
title: "[Q&A]: "
labels: ["q-and-a"]
assignees: []
body:
- type: checkboxes
id: prerequisites
attributes:
label: Prerequisites
description: Please confirm you have completed these steps before asking your question
options:
- label: I have searched existing discussions and issues to see if my question has been answered before
required: true
- label: I have read the [README](https://github.com/alexta69/metube#readme) and relevant sections
required: true
- label: I have checked the [Wiki](https://github.com/alexta69/metube/wiki) for configuration examples
required: true
- type: markdown
attributes:
value: |
## Quick Resources
Before asking your question, please check these resources:
- **[README](https://github.com/alexta69/metube#readme)** - Complete setup and configuration guide
- **[YTDL_OPTIONS Cookbook](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook)** - Common yt-dlp configurations
- **[OUTPUT_TEMPLATE Cookbook](https://github.com/alexta69/metube/wiki/OUTPUT_TEMPLATE-Cookbook)** - Filename template examples
- type: textarea
id: question
attributes:
label: Your Question
description: What would you like to know about MeTube?
placeholder: |
What are you trying to achieve?
What's your current setup?
What have you tried so far?
validations:
required: true
- type: textarea
id: additional-context
attributes:
label: Additional Context
description: Any other information that might be helpful (configuration, errors, screenshots, etc.)
placeholder: |
Configuration:
```yaml
# Your docker-compose.yml or environment variables
```
Error messages or logs:
[Paste any error messages or logs here]
Other relevant information:
[Screenshots, examples, etc.]
validations:
required: false
+10 -11
View File
@@ -9,24 +9,20 @@ body:
id: prerequisites
attributes:
label: Prerequisites
description: Please confirm you have completed these steps before submitting your bug report
description: Please confirm before submitting
options:
- label: I have searched existing issues and discussions to ensure this bug hasn't been reported before
required: true
- label: I have read the [troubleshooting section](https://github.com/alexta69/metube#-troubleshooting-and-submitting-issues) in the README
required: true
- label: I have tested this issue with yt-dlp directly (not just through MeTube UI) as described in the README
required: true
- label: I have checked that this is not a yt-dlp issue (if it is, please report it to [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues) instead)
- label: If the download itself fails, I have tested the same URL and options with yt-dlp directly (see the note below) — or this bug is not download-related
required: true
- type: markdown
attributes:
value: |
## Important Notes
- **MeTube is only a UI for yt-dlp** - issues with authentication, postprocessing, permissions, or other yt-dlp functionality should be reported to the [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues)
- Before reporting, please test with yt-dlp directly using: `docker exec -ti metube sh` then `cd /downloads` and run yt-dlp commands
- If yt-dlp works directly but MeTube doesn't, then it's a MeTube issue
- **MeTube is only a UI for [yt-dlp](https://github.com/yt-dlp/yt-dlp)** issues with authentication, postprocessing, site support, or other download functionality should be reported to the [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues)
- To test with yt-dlp directly: `docker exec -ti metube sh`, then `cd /downloads` and run `yt-dlp` with your URL and options
- If yt-dlp works directly but MeTube doesn't, then it's a MeTube issue — report it here
- type: textarea
id: bug-description
@@ -47,10 +43,13 @@ body:
id: ytdl-test-results
attributes:
label: yt-dlp Direct Test Results
description: Results of testing the same URL/configuration directly with yt-dlp (required)
description: >-
If the download itself fails: paste the exact yt-dlp command you ran AND its output —
download-failure reports without this will be closed as needs-info.
If this is a UI or app bug that doesn't involve a failing download, just write "UI bug".
placeholder: |
Command used: yt-dlp [your-command-here]
Result: [success/error and output]
Result: [paste the output here]
validations:
required: true
+7 -4
View File
@@ -1,8 +1,11 @@
blank_issues_enabled: false
contact_links:
- name: MeTube Community Discussions
url: https://github.com/alexta69/metube/discussions
about: Ask questions and discuss MeTube with the community
- name: yt-dlp Issues
- name: ❓ Questions & Support
url: https://github.com/alexta69/metube/discussions/categories/q-a
about: Ask usage and configuration questions in Discussions Q&A — issues are for bugs and feature requests
- name: 📖 Troubleshooting FAQ
url: https://github.com/alexta69/metube/wiki/Troubleshooting-FAQ
about: Common problems and their fixes
- name: ⬇️ yt-dlp Issues
url: https://github.com/yt-dlp/yt-dlp/issues
about: Report issues related to video downloading, authentication, or site support
+22 -15
View File
@@ -5,28 +5,35 @@ labels: ["enhancement"]
assignees: []
body:
- type: markdown
attributes:
value: |
## Read this first — MeTube's scope
MeTube's scope is deliberately narrow: **it downloads well and stops once the file is written.**
Features that improve the download itself are welcome — usually as community PRs, since the
maintainer is unlikely to implement requests ([details](https://github.com/alexta69/metube#-submitting-feature-requests)).
Post-download file management is out of scope regardless of implementation quality.
**Already decided — please don't re-request:**
- 🔒 Built-in authentication / password → use a reverse proxy ([#931](https://github.com/alexta69/metube/issues/931), [wiki](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations))
- 🔔 Notifications on download completion → available today via the `Exec` postprocessor ([recipe](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook))
- ✏️ Renaming, converting, tagging, or organizing files after download → out of scope ([#495](https://github.com/alexta69/metube/issues/495), [#1027](https://github.com/alexta69/metube/issues/1027))
- 🎚️ Pre-download format/quality/audio-track picker → tracked in [#1032](https://github.com/alexta69/metube/issues/1032)
- 📋 Pre-download playlist item browser → tracked in [#1030](https://github.com/alexta69/metube/issues/1030)
- 🎛️ Config switches to disable individual features → declined ([#976](https://github.com/alexta69/metube/issues/976))
- ⏰ Download scheduler → declined; add items with auto-start off and start them when you like ([#838](https://github.com/alexta69/metube/issues/838))
- type: checkboxes
id: prerequisites
attributes:
label: Prerequisites
description: Please confirm you have completed these steps before submitting your feature request
description: Please confirm before submitting
options:
- label: I have searched existing issues and discussions to ensure this feature hasn't been requested before
- label: I have searched existing issues and checked the "Already decided" list above
required: true
- label: I have read the [feature request guidelines](https://github.com/alexta69/metube#-submitting-feature-requests) in the README
- label: I have considered if this feature belongs in yt-dlp instead (downloading, processing, site support) — if so, please request it in the [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues)
required: true
- label: I understand that MeTube development relies on community contributions and the maintainer is not likely to implement this feature
required: true
- label: I have considered if this feature should be implemented in yt-dlp instead of MeTube (if so, please report to [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues))
required: true
- type: markdown
attributes:
value: |
## Important Notes
- **MeTube development relies on code contributions by the community** - the project is feature-complete for the maintainer's use cases
- **Consider if this belongs in yt-dlp** - if it's related to video downloading, processing, or site support, it might belong in the [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues) instead
- **Some features may not be accepted** - in an effort to reduce bloat, some PRs may not be accepted
- type: textarea
id: feature-description
-62
View File
@@ -1,62 +0,0 @@
name: ❓ Question
description: Ask a question about MeTube usage, configuration, or general help
title: "[Question]: "
labels: ["question"]
assignees: []
body:
- type: checkboxes
id: prerequisites
attributes:
label: Prerequisites
description: Please confirm you have completed these steps before asking your question
options:
- label: I have searched existing issues and discussions to see if my question has been answered before
required: true
- label: I have read the [README](https://github.com/alexta69/metube#readme) and relevant sections
required: true
- label: I have checked the [Wiki](https://github.com/alexta69/metube/wiki) for configuration examples
required: true
- label: I have read the [troubleshooting section](https://github.com/alexta69/metube#-troubleshooting-and-submitting-issues) if this is a technical issue
required: true
- type: markdown
attributes:
value: |
## Quick Resources
Before asking your question, please check these resources:
- **[README](https://github.com/alexta69/metube#readme)** - Complete setup and configuration guide
- **[YTDL_OPTIONS Cookbook](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook)** - Common yt-dlp configurations
- **[OUTPUT_TEMPLATE Cookbook](https://github.com/alexta69/metube/wiki/OUTPUT_TEMPLATE-Cookbook)** - Filename template examples
- type: textarea
id: question
attributes:
label: Your Question
description: What would you like to know about MeTube?
placeholder: |
What are you trying to achieve?
What's your current setup?
What have you tried so far?
validations:
required: true
- type: textarea
id: additional-context
attributes:
label: Additional Context
description: Any other information that might be helpful (configuration, errors, screenshots, etc.)
placeholder: |
Configuration:
```yaml
# Your docker-compose.yml or environment variables
```
Error messages or logs:
[Paste any error messages or logs here]
Other relevant information:
[Screenshots, examples, etc.]
validations:
required: false
+26 -40
View File
@@ -117,25 +117,27 @@ jobs:
- name: Get current date
id: date
run: echo "date=$(date +'%Y.%m.%d')" >> $GITHUB_OUTPUT
- name: Checkout
uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Get commits since last release
id: commits
env:
DATE: ${{ steps.date.outputs.date }}
run: |
# Fetch all tags
git fetch --tags
# Get the last tag (sorted by version, using date format YYYY.MM.DD)
LAST_TAG=$(git tag -l --sort=-version:refname | grep -E '^[0-9]{4}\.[0-9]{2}\.[0-9]{2}$' | head -n 1)
# Exclude today's tag: on a same-day rerun the notes must cover the
# whole day, not just the commits since the morning release.
LAST_TAG=$(git tag -l --sort=-version:refname | grep -E '^[0-9]{4}\.[0-9]{2}\.[0-9]{2}$' | grep -v "^${DATE}$" | head -n 1)
if [ -z "$LAST_TAG" ]; then
# No previous release, skip commits for first release
COMMITS=""
echo "has_commits=false" >> $GITHUB_OUTPUT
else
# Get commits since last tag
COMMITS=$(git log ${LAST_TAG}..HEAD --pretty=format:"- %s (%h)" --no-merges)
if [ -z "$COMMITS" ]; then
echo "has_commits=false" >> $GITHUB_OUTPUT
@@ -144,17 +146,12 @@ jobs:
fi
fi
# Escape for use in YAML/multiline output
{
echo 'commits<<EOF'
echo "$COMMITS"
echo EOF
} >> $GITHUB_OUTPUT
# Also output for debugging
echo "Last tag: ${LAST_TAG:-none}"
echo "Commits since last release:"
echo "$COMMITS"
- name: Generate release body
id: release_body
env:
@@ -185,38 +182,27 @@ jobs:
fi
} > release_body.txt
{
echo 'body<<EOF'
cat release_body.txt
echo EOF
} >> $GITHUB_OUTPUT
- name: Delete existing release if present
- name: Create or update GitHub Release (mark as latest)
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
TAG_NAME: ${{ steps.date.outputs.date }}
run: |
# Check if release exists and delete it
if gh release view "$TAG_NAME" &>/dev/null; then
echo "Release $TAG_NAME already exists, deleting it..."
gh release delete "$TAG_NAME" --yes || true
if gh release view "$TAG_NAME" >/dev/null 2>&1; then
echo "Release $TAG_NAME exists; updating."
# Force-move the tag in place so it matches the rebuilt Docker
# image. Never delete+recreate the tag: that corrupts GitHub's
# release index (broke release 2026.07.05).
gh api -X PATCH "repos/${GITHUB_REPOSITORY}/git/refs/tags/${TAG_NAME}" \
-f sha="$GITHUB_SHA" -F force=true
gh release edit "$TAG_NAME" \
--title "Release $TAG_NAME" \
--notes-file release_body.txt \
--latest
else
echo "Release $TAG_NAME does not exist; creating."
gh release create "$TAG_NAME" \
--target "$GITHUB_SHA" \
--title "Release $TAG_NAME" \
--notes-file release_body.txt \
--latest
fi
# Fetch tags to check remote
git fetch --tags
# Check if tag exists (locally or remotely) and delete it
if git rev-parse "$TAG_NAME" &>/dev/null 2>&1 || git ls-remote --tags origin "$TAG_NAME" | grep -q "$TAG_NAME"; then
echo "Tag $TAG_NAME already exists, deleting it..."
git tag -d "$TAG_NAME" 2>/dev/null || true
git push origin ":refs/tags/$TAG_NAME" || true
fi
- name: Create GitHub Release
uses: softprops/action-gh-release@v3
with:
tag_name: ${{ steps.date.outputs.date }}
name: Release ${{ steps.date.outputs.date }}
body_path: release_body.txt
draft: false
prerelease: false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+21
View File
@@ -0,0 +1,21 @@
name: readme-size
on:
push:
paths:
- 'README.md'
pull_request:
paths:
- 'README.md'
jobs:
check-size:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v7
- name: Check README stays under Docker Hub's 25k character limit
run: |
size=$(wc -c < README.md)
echo "README.md is ${size} bytes (limit 25000)"
test "$size" -lt 25000
+97
View File
@@ -1,5 +1,53 @@
# Agent Guidelines
## Project scope — read this before planning a feature
MeTube's contract is: give it a URL, it runs yt-dlp well, and correct files appear.
The maintainer holds a deliberate line on what belongs inside that contract, and PRs
on the wrong side of it are declined **regardless of code quality**. Check your plan
against this line before writing any code.
**In scope — improving the write itself:**
- Features that make the file yt-dlp writes at download time come out more correct,
using only data the extractor already provides (e.g. filling a missing album-artist
tag from the extractor's own metadata).
- Surfacing functionality yt-dlp itself owns and maintains as first-class UI options
(e.g. a SponsorBlock toggle that just passes yt-dlp postprocessor params).
- Download queue, subscriptions, output templates, and UI improvements to the
download workflow.
**Out of scope — managing files after they exist:**
- Tag editors, metadata dialogs, or any workflow that rewrites files after the
download has finished. This holds even for slimmed-down versions.
- Lookups against external metadata services (iTunes, Deezer, MusicBrainz, etc.).
More broadly: any new dependency on a third-party API, or new network egress from
self-hosted instances, beyond what yt-dlp itself performs.
- Library organization: moving/renaming existing files into Artist/Album layouts,
watch-folder processing, and similar media-manager features. Dedicated tools
(beets, MusicBrainz Picard, Lidarr) do this properly; the README points users
to them.
**Corollaries that shape borderline PRs:**
- Site-specific intelligence (parsing playlist-ID prefixes, URL path conventions,
and other platform internals) is extractor work and belongs upstream in yt-dlp,
not re-implemented here — it silently breaks when the platform changes and
MeTube would own the breakage.
- Prefer enriching yt-dlp's info dict and letting its existing pipeline
(FFmpegMetadata etc.) do the writing, over adding custom per-format tag-writing
code to MeTube.
- Supplemental processing must never fail a download that otherwise succeeded:
warn and continue, don't raise.
- Keep feature scope minimal on first submission. A hardcoded sensible default
beats a configuration surface; follow-ups can add options when users actually
ask. PRs that bundle several "reasonable next steps" invite rejection of the
whole.
If a feature idea fails this test, the accepted alternative is usually a README
section documenting how to pair MeTube with the right dedicated tool.
## README.md size constraint
The README.md is synced to Docker Hub, which has a **25,000 character limit**.
@@ -30,6 +78,24 @@ uv run pytest app/tests/
All of these run in CI (`.github/workflows/main.yml`) on every push to master and must pass.
Gotchas:
- Backend tests must run **from the repo root**: `main.py` resolves the static-assets
path relative to the cwd, and several test modules import `main`. Running from
`app/` makes five test modules fail to import.
- The frontend must be **built before** running backend tests (same reason — the
assets at `ui/dist/metube/browser` must exist). The command order above is
load-bearing.
- `app/tests/test_ytdl_utils.py` stubs `yt_dlp` at import time. Run standalone,
two tests fail with `AttributeError: <module 'yt_dlp'> does not have the
attribute 'YoutubeDL'`; under the full suite the real module is imported first
and they pass. This is a known quirk, not a bug to fix in the code under test.
Every non-markdown push to master builds multi-arch Docker images and cuts a dated
release the same day. **Master is continuously released** — a PR must be
release-ready exactly as merged; there is no stabilization window for follow-up
fixes.
## Code style
Follow `.editorconfig`:
@@ -56,5 +122,36 @@ ui/src/app/ — Angular standalone components (no NgModules)
- Backend configuration lives in the `Config` class in `app/main.py` with env-var defaults in `_DEFAULTS`. New env vars go there.
- Real-time communication uses Socket.IO events, not REST polling.
- Frontend uses standalone Angular components with `inject()` for DI, RxJS Subjects for state, and `takeUntilDestroyed()` for cleanup.
- Frontend components use OnPush change detection: subscribe callbacks must call `cdr.markForCheck()`.
- State is persisted as JSON files via `AtomicJsonStore` in `app/state_store.py`.
- Persisted state stays compact: the completed queue deliberately drops bulky entry data (see `_compact_persisted_entry` in `app/ytdl.py`). Don't expand what gets persisted without discussion.
- Custom yt-dlp postprocessors added to `ytdl_params['postprocessors']` run in **list order** within a stage. When combining postprocessors, mirror the ordering the yt-dlp CLI would produce (e.g. sponsor-segment removal before chapter splitting).
- No pre-commit hooks — linting and tests are enforced in CI only.
## Checklist: adding a per-download option
New options on the download form (the `split_by_chapters` pattern) need **all** of
these pieces — the last three are the ones commonly missed:
1. `parse_download_options` in `app/main.py`.
2. A field on `DownloadInfo` in `app/ytdl.py`.
3. A `hasattr` backfill in `DownloadInfo.__setstate__` for old persisted records.
4. The safe-deserialization field list in `app/ytdl.py`.
5. UI form control + cookie persistence in `ui/src/app/app.ts` / `app.html`, and
the payload in `downloads.service.ts` (plus its spec).
6. The redownload path in `app.ts`, so retries carry the option.
7. If the option makes sense for unattended downloads: threading through
`app/subscriptions.py` (`SubscriptionInfo` field, serializer, add/update
routes, the enqueue call) — or a note in the PR that it's deliberately
direct-downloads-only.
## Security invariants
User input and extractor-provided metadata (titles, playlist names, URLs) are
untrusted. Use the existing guards instead of hand-rolling:
- User-submitted URLs go through the SSRF guard (see `test_url_guard.py` for the
expected behavior).
- Anything that becomes a filesystem path goes through `_is_within_directory` and
`_sanitize_path_component` in `app/ytdl.py` — including values that arrive via
yt-dlp metadata, which sites can influence.
+33 -88
View File
@@ -3,12 +3,12 @@
![Build Status](https://github.com/alexta69/metube/actions/workflows/main.yml/badge.svg)
![Docker Pulls](https://img.shields.io/docker/pulls/alexta69/metube.svg)
MeTube is a self-hosted web UI for `yt-dlp`, for downloading media from YouTube and [dozens of other sites](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md).
MeTube is a self-hosted web UI for `yt-dlp`, for downloading media from YouTube and [dozens of other sites](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md). Docker images are multi-arch (amd64/arm64).
Key capabilities:
* Download videos, audio, captions, and thumbnails from a browser UI.
* Download playlists and channels, with configurable output and download options.
* Subscribe to channels and playlists, periodically check for new items, and queue new uploads automatically.
* [Subscribe](https://github.com/alexta69/metube/wiki/Subscriptions) to channels and playlists, periodically check for new items, and queue new uploads automatically.
![screenshot1](https://github.com/alexta69/metube/raw/master/screenshot.gif)
@@ -18,7 +18,7 @@ Key capabilities:
docker run -d -p 8081:8081 -v /path/to/downloads:/downloads ghcr.io/alexta69/metube
```
## 🐳 Run using docker-compose
## 🐳 Run using Docker Compose
```yaml
services:
@@ -34,7 +34,16 @@ services:
## ⚙️ Configuration via environment variables
Certain values can be set via environment variables, using the `-e` parameter on the docker command line, or the `environment:` section in docker-compose.
Certain values can be set via environment variables, using the `-e` parameter on the docker command line, or the `environment:` section in Docker Compose.
### 🏠 Runtime & Permissions
* __PUID__: User under which MeTube will run. Defaults to `1000` (legacy `UID` also supported).
* __PGID__: Group under which MeTube will run. Defaults to `1000` (legacy `GID` also supported).
* __UMASK__: Umask value used by MeTube. Defaults to `022`.
* __DEFAULT_THEME__: Default theme to use for the UI, can be set to `light`, `dark`, or `auto`. Defaults to `auto`.
* __LOGLEVEL__: Log level, can be set to `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, or `NONE`. Defaults to `INFO`.
* __ENABLE_ACCESSLOG__: Whether to enable access log. Defaults to `false`.
### ⬇️ Download Behavior
@@ -71,6 +80,7 @@ Certain values can be set via environment variables, using the `-e` parameter on
* __YTDL_OPTIONS_PRESETS__: Named bundles of yt-dlp options, selectable per download in the UI. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options) for format and examples.
* __YTDL_OPTIONS_PRESETS_FILE__: Path to a JSON file containing presets. Monitored and reloaded automatically on changes. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options).
* __ALLOW_YTDL_OPTIONS_OVERRIDES__: Whether to show a free-text field in the UI for per-download yt-dlp option overrides. Defaults to `false`. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options) for details and security considerations.
* __ALLOW_PRIVATE_ADDRESSES__: Whether to allow downloads from private, loopback, link-local and other non-global addresses. Defaults to `false`, which protects against SSRF by refusing URLs that resolve to internal hosts. Set to `true` only in trusted environments — for example when routing traffic through a proxy/VPN client in Fake-IP mode (sing-box, Clash, Mihomo), which resolves hosts to the `198.18.0.0/15` range. Enabling this disables the SSRF protection entirely, so only use it when you control the network.
* __YTDL_NIGHTLY_UPDATE_TIME__: If set, will cause MeTube to use [nightly yt-dlp builds](https://github.com/yt-dlp/yt-dlp-nightly-builds) instead of the stable releases. Set to the time (`HH:MM`, 24-hour) when you want the daily upgrades and MeTube restart to happen. Defaults to empty (disabled).
### 🌐 Web Server & URLs
@@ -83,18 +93,9 @@ Certain values can be set via environment variables, using the `-e` parameter on
* __HTTPS__: Use `https` instead of `http` (__CERTFILE__ and __KEYFILE__ required). Defaults to `false`.
* __CERTFILE__: HTTPS certificate file path.
* __KEYFILE__: HTTPS key file path.
* __CORS_ALLOWED_ORIGINS__: Comma-separated list of origins permitted to make cross-origin requests to the MeTube API. When unset or empty, all cross-origin requests are denied. Set to `*` to allow all origins. This must be configured for [browser extensions](#-browser-extensions), [bookmarklets](#-bookmarklet), and any other browser-based tools that contact MeTube from a different origin. For browser extensions use `*` (see below); for bookmarklets you can list specific sites, e.g. `https://www.youtube.com,https://www.vimeo.com`.
* __CORS_ALLOWED_ORIGINS__: Comma-separated list of origins permitted to make cross-origin requests to the MeTube API; `*` allows all. When unset or empty, all cross-origin requests are denied. Required for browser extensions and bookmarklets — see [Sending links to MeTube](#-sending-links-to-metube).
* __ROBOTS_TXT__: A path to a `robots.txt` file mounted in the container.
### 🏠 Basic Setup
* __PUID__: User under which MeTube will run. Defaults to `1000` (legacy `UID` also supported).
* __PGID__: Group under which MeTube will run. Defaults to `1000` (legacy `GID` also supported).
* __UMASK__: Umask value used by MeTube. Defaults to `022`.
* __DEFAULT_THEME__: Default theme to use for the UI, can be set to `light`, `dark`, or `auto`. Defaults to `auto`.
* __LOGLEVEL__: Log level, can be set to `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, or `NONE`. Defaults to `INFO`.
* __ENABLE_ACCESSLOG__: Whether to enable access log. Defaults to `false`.
## 🎛️ Configuring yt-dlp options
MeTube lets you customize how [yt-dlp](https://github.com/yt-dlp/yt-dlp) behaves at three levels, from broadest to most specific:
@@ -229,47 +230,27 @@ In case you need to use your browser's cookies with MeTube, for example to downl
* After upload, the cookie indicator should show as active.
* Use **Delete Cookies** in the same section to remove uploaded cookies.
## 🔌 Browser extensions
## 🔗 Sending links to MeTube
Browser extensions allow right-clicking videos and sending them directly to MeTube. If you're on an HTTPS page, your MeTube instance must be behind an HTTPS reverse proxy (see below) for extensions to work.
Several integrations let you send URLs to MeTube from wherever you are, instead of pasting them into the UI. The browser-based ones make cross-origin requests, so they require `CORS_ALLOWED_ORIGINS` to be set; and if you're on an HTTPS page, your MeTube instance must be served over HTTPS too (with `HTTPS=true` or behind an HTTPS reverse proxy see below).
Since browser extensions make requests from their own origin (`chrome-extension://...` or `moz-extension://...`), you must set `CORS_ALLOWED_ORIGINS=*` for them to work.
__Browser extensions__ allow right-clicking videos and sending them directly to MeTube. Since extensions request from their own origin, set `CORS_ALLOWED_ORIGINS=*`.
* __Chrome:__ contributed by [Rpsl](https://github.com/rpsl) — install from the [Chrome Webstore](https://chrome.google.com/webstore/detail/metube-downloader/fbmkmdnlhacefjljljlbhkodfmfkijdh) or [from sources](https://github.com/Rpsl/metube-browser-extension).
* __Firefox:__ contributed by [nanocortex](https://github.com/nanocortex) — install from [Firefox Addons](https://addons.mozilla.org/en-US/firefox/addon/metube-downloader) or get sources [here](https://github.com/nanocortex/metube-firefox-addon).
__Chrome:__ contributed by [Rpsl](https://github.com/rpsl). You can install it from [Google Chrome Webstore](https://chrome.google.com/webstore/detail/metube-downloader/fbmkmdnlhacefjljljlbhkodfmfkijdh) or use developer mode and install [from sources](https://github.com/Rpsl/metube-browser-extension).
__Bookmarklets__ send the currently open page to MeTube with one click. Add the origins of the sites where you use them to `CORS_ALLOWED_ORIGINS`, e.g. `https://www.youtube.com,https://www.vimeo.com`. The code (Chrome and Firefox variants, contributed by [kushfest](https://github.com/kushfest) and [shoonya75](https://github.com/shoonya75)) is in the [Bookmarklets wiki page](https://github.com/alexta69/metube/wiki/Bookmarklets).
__Firefox:__ contributed by [nanocortex](https://github.com/nanocortex). You can install it from [Firefox Addons](https://addons.mozilla.org/en-US/firefox/addon/metube-downloader) or get sources from [here](https://github.com/nanocortex/metube-firefox-addon).
__iOS Shortcut:__ [rithask](https://github.com/rithask) created an [iOS shortcut](https://www.icloud.com/shortcuts/66627a9f334c467baabdb2769763a1a6) for sending URLs to MeTube from Safari's share menu; it prompts for your instance address on first use.
## 📱 iOS Shortcut
__Raycast:__ [dotvhs](https://github.com/dotvhs) has created an [extension for Raycast](https://www.raycast.com/dot/metube) for adding videos to MeTube directly from Raycast.
[rithask](https://github.com/rithask) created an iOS shortcut to send URLs to MeTube from Safari. Enter the MeTube instance address when prompted which will be saved for later use. You can run the shortcut from Safaris share menu. The shortcut can be downloaded from [this iCloud link](https://www.icloud.com/shortcuts/66627a9f334c467baabdb2769763a1a6).
## 🎵 Pairing with a music tagger
## 🔖 Bookmarklet
MeTube deliberately stops once the file is written — tagging and library organization belong to dedicated tools. Point one at your audio download folder (`AUDIO_DOWNLOAD_DIR`):
[kushfest](https://github.com/kushfest) has created a Chrome bookmarklet for sending the currently open webpage to MeTube. Please note that if you're on an HTTPS page, your MeTube instance must be configured with `HTTPS` as `true` in the environment, or be behind an HTTPS reverse proxy (see below) for the bookmarklet to work.
Since bookmarklets run in the context of the current page (e.g. youtube.com), the requests they make to MeTube are cross-origin. You must add the origins of sites where you use the bookmarklet to the __CORS_ALLOWED_ORIGINS__ environment variable, otherwise the browser will block the requests. For example, to use the bookmarklet on YouTube and Vimeo: `CORS_ALLOWED_ORIGINS=https://www.youtube.com,https://www.vimeo.com`.
GitHub doesn't allow embedding JavaScript as a link, so the bookmarklet has to be created manually by copying the following code to a new bookmark you create on your bookmarks bar. Change the hostname in the URL below to point to your MeTube instance.
```javascript
javascript:!function(){xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.withCredentials=true;xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function(){if(xhr.status==200){alert("Sent to metube!")}else{alert("Send to metube failed. Check the javascript console for clues.")}}}();
```
[shoonya75](https://github.com/shoonya75) has contributed a Firefox version:
```javascript
javascript:(function(){xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function(){if(xhr.status==200){alert("Sent to metube!")}else{alert("Send to metube failed. Check the javascript console for clues.")}}})();
```
The above bookmarklets use `alert()` for notifications. This variant shows a toast instead (Chrome — for Firefox, replace the `!function(){...}()` wrapper with `(function(){...})()`):
```javascript
javascript:!function(){function notify(msg) {var sc = document.scrollingElement.scrollTop; var text = document.createElement('span');text.innerHTML=msg;var ts = text.style;ts.all = 'revert';ts.color = '#000';ts.fontFamily = 'Verdana, sans-serif';ts.fontSize = '15px';ts.backgroundColor = 'white';ts.padding = '15px';ts.border = '1px solid gainsboro';ts.boxShadow = '3px 3px 10px';ts.zIndex = '100';document.body.appendChild(text);ts.position = 'absolute'; ts.top = 50 + sc + 'px'; ts.left = (window.innerWidth / 2)-(text.offsetWidth / 2) + 'px'; setTimeout(function () { text.style.visibility = "hidden"; }, 1500);}xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function() { if(xhr.status==200){notify("Sent to metube!")}else {notify("Send to metube failed. Check the javascript console for clues.")}}}();
```
## ⚡ Raycast extension
[dotvhs](https://github.com/dotvhs) has created an [extension for Raycast](https://www.raycast.com/dot/metube) for adding videos to MeTube directly from Raycast.
* [beets](https://beets.io) — `beet import` matches tracks against MusicBrainz, fixes tags, and files them into an Artist/Album library; headless and scriptable.
* [MusicBrainz Picard](https://picard.musicbrainz.org) — GUI tagger with acoustic fingerprinting.
* [Lidarr](https://lidarr.audio) — full music library manager; add the folder as an import path.
## 🔒 HTTPS support, and running behind a reverse proxy
@@ -293,11 +274,7 @@ services:
- KEYFILE=/ssl/key.pem
```
MeTube can also run behind a reverse proxy for HTTPS termination or authentication. When serving under a subdirectory, set `URL_PREFIX` accordingly.
The [linuxserver/swag](https://docs.linuxserver.io/general/swag) image includes ready-made snippets for MeTube in [subfolder](https://github.com/linuxserver/reverse-proxy-confs/blob/master/metube.subfolder.conf.sample) and [subdomain](https://github.com/linuxserver/reverse-proxy-confs/blob/master/metube.subdomain.conf.sample) modes, plus Authelia for authentication.
### 🌐 NGINX
MeTube can also run behind a reverse proxy for HTTPS termination or authentication. When serving under a subdirectory, set `URL_PREFIX` accordingly. MeTube uses WebSocket for real-time updates, so the proxy must pass the `Upgrade`/`Connection` headers, as in this NGINX example:
```nginx
location /metube/ {
@@ -309,41 +286,7 @@ location /metube/ {
}
```
Note: the extra `proxy_set_header` directives are there to make WebSocket work.
### 🌐 Apache
Contributed by [PIE-yt](https://github.com/PIE-yt). Source [here](https://gist.github.com/PIE-yt/29e7116588379032427f5bd446b2cac4).
```apache
# For putting in your Apache sites site.conf
# Serves MeTube under a /metube/ subdir (http://yourdomain.com/metube/)
<Location /metube/>
ProxyPass http://localhost:8081/ retry=0 timeout=30
ProxyPassReverse http://localhost:8081/
</Location>
<Location /metube/socket.io>
RewriteEngine On
RewriteCond %{QUERY_STRING} transport=websocket [NC]
RewriteRule /(.*) ws://localhost:8081/socket.io/$1 [P,L]
ProxyPass http://localhost:8081/socket.io retry=0 timeout=30
ProxyPassReverse http://localhost:8081/socket.io
</Location>
```
### 🌐 Caddy
The following example Caddyfile gets a reverse proxy going behind [caddy](https://caddyserver.com).
```caddyfile
example.com {
route /metube/* {
uri strip_prefix metube
reverse_proxy metube:8081
}
}
```
Apache, Caddy, and [linuxserver/swag](https://docs.linuxserver.io/general/swag) (with Authelia) examples are in the [Reverse proxy configurations wiki page](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations).
## 🔄 Updating yt-dlp
@@ -358,9 +301,11 @@ docker exec -ti metube sh
cd /downloads
```
Common issues and their fixes are collected in the [Troubleshooting FAQ](https://github.com/alexta69/metube/wiki/Troubleshooting-FAQ) on the wiki.
## 💡 Submitting feature requests
MeTube development relies on community contributions. If you need additional features, please submit a PR. Create an issue first to discuss the implementation — some PRs may not be accepted to reduce bloat. Feature requests without an accompanying PR are unlikely to be fulfilled.
MeTube development relies on community contributions. If you need additional features, please submit a PR. Create an issue first to discuss the implementation before writing code — MeTube's scope is deliberately narrow: it downloads well and stops once the file is written. Features that improve the download itself are welcome; post-download file management (tag editing, metadata lookups, library organization) is out of scope regardless of implementation quality — see [AGENTS.md](AGENTS.md) for the full policy. Feature requests without an accompanying PR are unlikely to be fulfilled.
## 🛠️ Building and running locally
+24
View File
@@ -0,0 +1,24 @@
# Security Policy
## Reporting a vulnerability
Please report vulnerabilities privately via
[GitHub private vulnerability reporting](https://github.com/alexta69/metube/security/advisories/new)
(Security tab → "Report a vulnerability"). Do **not** open a public issue for
security problems.
You can expect an initial response within a few days. Please include a
reproduction and the MeTube release version (visible in the UI footer).
## Supported versions
MeTube is continuously released; only the **latest release** is supported.
Update to the current Docker image before reporting.
## Scope notes
MeTube ships **without authentication by design** — it is intended to run on a
trusted network or behind an authenticating reverse proxy (see the
[wiki](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations)).
Reports that reduce to "the UI is reachable without a login" are expected
behavior, not vulnerabilities.
+2 -5
View File
@@ -81,6 +81,7 @@ class Config:
'YTDL_OPTIONS_PRESETS': '{}',
'YTDL_OPTIONS_PRESETS_FILE': '',
'ALLOW_YTDL_OPTIONS_OVERRIDES': 'false',
'ALLOW_PRIVATE_ADDRESSES': 'false',
'CORS_ALLOWED_ORIGINS': '',
'ROBOTS_TXT': '',
'HOST': '0.0.0.0',
@@ -96,7 +97,7 @@ class Config:
'YTDL_NIGHTLY_UPDATE_TIME': '',
}
_BOOLEAN = ('DOWNLOAD_DIRS_INDEXABLE', 'CUSTOM_DIRS', 'CREATE_CUSTOM_DIRS', 'DELETE_FILE_ON_TRASHCAN', 'HTTPS', 'ENABLE_ACCESSLOG', 'ALLOW_YTDL_OPTIONS_OVERRIDES')
_BOOLEAN = ('DOWNLOAD_DIRS_INDEXABLE', 'CUSTOM_DIRS', 'CREATE_CUSTOM_DIRS', 'DELETE_FILE_ON_TRASHCAN', 'HTTPS', 'ENABLE_ACCESSLOG', 'ALLOW_YTDL_OPTIONS_OVERRIDES', 'ALLOW_PRIVATE_ADDRESSES')
def __init__(self):
for k, v in self._DEFAULTS.items():
@@ -717,8 +718,6 @@ def parse_download_options(post: dict) -> dict:
if custom_name_prefix is None:
custom_name_prefix = ''
if custom_name_prefix and ('..' in custom_name_prefix or custom_name_prefix.startswith('/') or custom_name_prefix.startswith('\\')):
raise web.HTTPBadRequest(reason='custom_name_prefix must not contain ".." or start with a path separator')
if auto_start is None:
auto_start = True
if playlist_item_limit is None:
@@ -743,8 +742,6 @@ def parse_download_options(post: dict) -> dict:
enabled=config.ALLOW_YTDL_OPTIONS_OVERRIDES,
)
if chapter_template and ('..' in chapter_template or chapter_template.startswith('/') or chapter_template.startswith('\\')):
raise web.HTTPBadRequest(reason='chapter_template must not contain ".." or start with a path separator')
if not SUBTITLE_LANGUAGE_RE.fullmatch(subtitle_language):
raise web.HTTPBadRequest(reason='subtitle_language must match pattern [A-Za-z0-9-] and be at most 35 characters')
if subtitle_mode not in VALID_SUBTITLE_MODES:
+11
View File
@@ -19,6 +19,7 @@ import yt_dlp.networking.impersonate
import bg_tasks
from dl_formats import merge_ytdl_option_layers
from state_store import AtomicJsonStore, read_legacy_shelf
from url_guard import validate_url
log = logging.getLogger("subscriptions")
@@ -113,6 +114,9 @@ def extract_flat_playlist(
nested_url = _entry_video_url(ent)
if not nested_url:
continue
# nested_url comes from remote playlist content; guard it too.
if validate_url(nested_url, allow_private=getattr(config, "ALLOW_PRIVATE_ADDRESSES", False)) is not None:
continue
nested_info, nested_entries = extract_flat_playlist(
config,
nested_url,
@@ -542,6 +546,13 @@ class SubscriptionManager:
url = self._normalize_url(url)
if not url:
return {"status": "error", "msg": "Missing URL"}
# SSRF guard: block non-http(s) schemes and internal/metadata hosts
# before yt-dlp fetches the feed. May do a DNS lookup, so run off-loop.
url_error = await asyncio.get_running_loop().run_in_executor(
None, partial(validate_url, url, allow_private=getattr(self.config, "ALLOW_PRIVATE_ADDRESSES", False)))
if url_error is not None:
log.warning('Rejected subscription URL "%s": %s', url, url_error)
return {"status": "error", "msg": url_error}
try:
title_regex_stored = validate_title_regex(title_regex)
except re.error as exc:
-19
View File
@@ -138,25 +138,6 @@ async def test_add_invalid_subtitle_language(mock_dqueue):
await main.add(req)
@pytest.mark.asyncio
async def test_add_custom_name_prefix_path_traversal(mock_dqueue):
req = _json_request(_valid_video_add_body(custom_name_prefix="../evil"))
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
@pytest.mark.asyncio
async def test_add_chapter_template_path_traversal(mock_dqueue):
req = _json_request(
_valid_video_add_body(
split_by_chapters=True,
chapter_template="/etc/passwd%(title)s",
)
)
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
@pytest.mark.asyncio
async def test_add_invalid_json_body(mock_dqueue):
req = MagicMock(spec=web.Request)
+188
View File
@@ -0,0 +1,188 @@
"""Tests for the SSRF URL guard (``url_guard.validate_url``)."""
from __future__ import annotations
import socket
import unittest
from unittest import mock
import url_guard
from url_guard import (
validate_url,
_address_allowed_at_connect,
_guarded_getaddrinfo,
install_socket_guard,
)
def _addrinfo(*addrs, family=socket.AF_INET):
return [(family, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", (addr, 0)) for addr in addrs]
class NonUrlInputTests(unittest.TestCase):
"""Bare IDs and yt-dlp search/extractor prefixes must pass untouched."""
def test_bare_video_id_allowed(self):
self.assertIsNone(validate_url("dQw4w9WgXcQ"))
def test_ytsearch_prefix_allowed(self):
self.assertIsNone(validate_url("ytsearch:some song"))
def test_empty_string_allowed(self):
self.assertIsNone(validate_url(""))
def test_non_string_rejected(self):
self.assertIsNotNone(validate_url(None))
class SchemeTests(unittest.TestCase):
def test_file_scheme_blocked(self):
self.assertIsNotNone(validate_url("file:///etc/passwd"))
def test_ftp_scheme_blocked(self):
self.assertIsNotNone(validate_url("ftp://example.com/x"))
def test_data_scheme_blocked(self):
self.assertIsNotNone(validate_url("data://text/plain;base64,AAAA"))
class HostnameBlocklistTests(unittest.TestCase):
def test_localhost_blocked_without_lookup(self):
with mock.patch("url_guard.socket.getaddrinfo") as gai:
self.assertIsNotNone(validate_url("http://localhost:8080/x"))
gai.assert_not_called()
def test_localhost_subdomain_blocked(self):
self.assertIsNotNone(validate_url("http://foo.localhost/x"))
def test_gcp_metadata_name_blocked(self):
self.assertIsNotNone(validate_url("http://metadata.google.internal/x"))
class AddressResolutionTests(unittest.TestCase):
def _validate_with_addrs(self, url, *addrs, family=socket.AF_INET):
with mock.patch("url_guard.socket.getaddrinfo", return_value=_addrinfo(*addrs, family=family)):
return validate_url(url)
def test_public_https_allowed(self):
self.assertIsNone(self._validate_with_addrs("https://youtube.com/watch?v=x", "142.250.1.1"))
def test_public_http_allowed(self):
self.assertIsNone(self._validate_with_addrs("http://example.com/x", "93.184.216.34"))
def test_link_local_metadata_blocked(self):
self.assertIsNotNone(self._validate_with_addrs("http://metadata/x", "169.254.169.254"))
def test_loopback_ipv4_blocked(self):
self.assertIsNotNone(self._validate_with_addrs("http://127.0.0.1/x", "127.0.0.1"))
def test_private_rfc1918_blocked(self):
self.assertIsNotNone(self._validate_with_addrs("http://intranet/x", "10.0.0.5"))
def test_decimal_ip_form_blocked(self):
# 2852039166 == 169.254.169.254; the OS resolver normalizes it.
self.assertIsNotNone(self._validate_with_addrs("http://2852039166/x", "169.254.169.254"))
def test_ipv6_loopback_blocked(self):
self.assertIsNotNone(
self._validate_with_addrs("http://[::1]/x", "::1", family=socket.AF_INET6)
)
def test_ipv4_mapped_ipv6_metadata_blocked(self):
self.assertIsNotNone(
self._validate_with_addrs(
"http://evil/x", "::ffff:169.254.169.254", family=socket.AF_INET6
)
)
def test_mixed_public_and_private_blocked(self):
# If any resolved address is internal, reject the whole URL.
self.assertIsNotNone(self._validate_with_addrs("http://mixed/x", "142.250.1.1", "127.0.0.1"))
def test_resolution_failure_is_rejected(self):
# Fail closed: an unresolvable host cannot be verified as non-internal.
with mock.patch("url_guard.socket.getaddrinfo", side_effect=socket.gaierror):
self.assertIsNotNone(validate_url("http://does-not-resolve.example/x"))
class ConnectAddressPolicyTests(unittest.TestCase):
"""Connect-time policy: allow global + loopback, block everything else."""
def test_global_allowed(self):
self.assertTrue(_address_allowed_at_connect("142.250.1.1"))
def test_loopback_allowed(self):
# Loopback stays reachable so locally-configured proxies keep working.
self.assertTrue(_address_allowed_at_connect("127.0.0.1"))
self.assertTrue(_address_allowed_at_connect("::1"))
def test_link_local_metadata_blocked(self):
self.assertFalse(_address_allowed_at_connect("169.254.169.254"))
def test_private_blocked(self):
self.assertFalse(_address_allowed_at_connect("10.0.0.5"))
self.assertFalse(_address_allowed_at_connect("192.168.1.10"))
def test_ipv4_mapped_metadata_blocked(self):
self.assertFalse(_address_allowed_at_connect("::ffff:169.254.169.254"))
class GuardedGetaddrinfoTests(unittest.TestCase):
def test_internal_only_raises(self):
with mock.patch("url_guard._real_getaddrinfo", return_value=_addrinfo("169.254.169.254")):
with self.assertRaises(socket.gaierror):
_guarded_getaddrinfo("metadata", 80)
def test_filters_internal_keeps_global(self):
# Split-horizon rebinding: keep the public address, drop the internal one.
with mock.patch("url_guard._real_getaddrinfo", return_value=_addrinfo("142.250.1.1", "10.0.0.1")):
results = _guarded_getaddrinfo("mixed", 80)
self.assertEqual([r[4][0] for r in results], ["142.250.1.1"])
def test_loopback_passes(self):
with mock.patch("url_guard._real_getaddrinfo", return_value=_addrinfo("127.0.0.1")):
results = _guarded_getaddrinfo("localproxy", 9050)
self.assertEqual([r[4][0] for r in results], ["127.0.0.1"])
class AllowPrivateBypassTests(unittest.TestCase):
"""ALLOW_PRIVATE_ADDRESSES: trusted proxy/VPN environments opt out of the
SSRF address checks (e.g. Fake-IP clients that resolve to 198.18.0.0/15)."""
def test_internal_address_allowed_when_bypassed(self):
# Fake-IP benchmarking range that is_global rejects by default.
self.assertIsNone(validate_url("http://www.youtube.com/x", allow_private=True))
def test_private_host_allowed_when_bypassed(self):
# No DNS lookup needed: the bypass returns before resolution.
with mock.patch("url_guard.socket.getaddrinfo") as gai:
self.assertIsNone(validate_url("http://192.168.1.1/x", allow_private=True))
gai.assert_not_called()
def test_scheme_still_enforced_when_bypassed(self):
self.assertIsNotNone(validate_url("file:///etc/passwd", allow_private=True))
def test_socket_guard_not_installed_when_bypassed(self):
original = socket.getaddrinfo
try:
install_socket_guard(allow_private=True)
self.assertIs(socket.getaddrinfo, original)
finally:
socket.getaddrinfo = original
class InstallSocketGuardTests(unittest.TestCase):
def test_install_replaces_and_is_idempotent(self):
original = socket.getaddrinfo
try:
install_socket_guard()
self.assertIs(socket.getaddrinfo, url_guard._guarded_getaddrinfo)
# Re-installing must not wrap the wrapper (real fn captured at import).
install_socket_guard()
self.assertIs(socket.getaddrinfo, url_guard._guarded_getaddrinfo)
finally:
socket.getaddrinfo = original
if __name__ == "__main__":
unittest.main()
+330 -11
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import os
import pickle
import signal
import sys
@@ -15,6 +16,8 @@ from unittest.mock import MagicMock, patch
fake_yt_dlp = types.ModuleType("yt_dlp")
fake_networking = types.ModuleType("yt_dlp.networking")
fake_impersonate = types.ModuleType("yt_dlp.networking.impersonate")
fake_postprocessor = types.ModuleType("yt_dlp.postprocessor")
fake_postprocessor_common = types.ModuleType("yt_dlp.postprocessor.common")
fake_utils = types.ModuleType("yt_dlp.utils")
@@ -24,33 +27,192 @@ class _ImpersonateTarget:
return value
class _PostProcessor:
def __init__(self, downloader=None):
self._downloader = downloader
class _YoutubeDL:
"""Minimal stand-in so ``_ConfinedYoutubeDL`` can subclass it under the shim.
``prepare_filename`` is patched per-test; the real containment logic lives in
the ``_ConfinedYoutubeDL`` override, which is what the tests exercise.
"""
def __init__(self, params=None, **kwargs):
self.params = params or {}
def prepare_filename(self, *args, **kwargs):
return ""
def add_post_processor(self, *args, **kwargs):
pass
fake_utils.DownloadError = type("DownloadError", (Exception,), {})
fake_yt_dlp.YoutubeDL = _YoutubeDL
fake_impersonate.ImpersonateTarget = _ImpersonateTarget
fake_networking.impersonate = fake_impersonate
fake_postprocessor_common.PostProcessor = _PostProcessor
# The inner ``key`` group mirrors the real ``STR_FORMAT_RE_TMPL`` so that
# ``_OUTTMPL_FIELD_RE`` (compiled at import time) has the named group that
# ``_resolve_outtmpl_fields`` reads via ``match.group('key')``.
fake_utils.STR_FORMAT_RE_TMPL = r"(?P<prefix>)%\((?P<has_key>(?P<key>{}))\)(?P<format>[-0-9.]*{})"
fake_utils.STR_FORMAT_TYPES = "diouxXeEfFgGcrsa"
fake_yt_dlp.networking = fake_networking
fake_yt_dlp.postprocessor = fake_postprocessor
fake_yt_dlp.utils = fake_utils
sys.modules.setdefault("yt_dlp", fake_yt_dlp)
sys.modules.setdefault("yt_dlp.networking", fake_networking)
sys.modules.setdefault("yt_dlp.networking.impersonate", fake_impersonate)
sys.modules.setdefault("yt_dlp.postprocessor", fake_postprocessor)
sys.modules.setdefault("yt_dlp.postprocessor.common", fake_postprocessor_common)
sys.modules.setdefault("yt_dlp.utils", fake_utils)
import ytdl
from ytdl import (
Download,
DownloadInfo,
_compact_persisted_entry,
_convert_srt_to_txt_file,
_AlbumArtistPostProcessor,
_resolve_outtmpl_fields,
_sanitize_entry_for_pickle,
_sanitize_path_component,
)
# Detect whether the real yt-dlp is loaded (as opposed to the minimal fake
# shim above). _resolve_outtmpl_fields needs YoutubeDL at runtime.
_has_real_ytdlp = hasattr(sys.modules.get("yt_dlp"), "YoutubeDL")
# shim above). _resolve_outtmpl_fields needs YoutubeDL.evaluate_outtmpl at
# runtime, which the shim's YoutubeDL stand-in deliberately does not provide.
_has_real_ytdlp = hasattr(
getattr(sys.modules.get("yt_dlp"), "YoutubeDL", None), "evaluate_outtmpl"
)
class AlbumArtistPostProcessorTests(unittest.TestCase):
def setUp(self):
self.postprocessor = _AlbumArtistPostProcessor()
def test_fills_album_artist_from_artist(self):
info = {'album': 'CrasH Talk', 'artist': 'ScHoolboy Q'}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'ScHoolboy Q')
def test_uses_main_artist_for_featured_track(self):
info = {
'album': 'CrasH Talk',
'artists': ['ScHoolboy Q · Travis Scott'],
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'ScHoolboy Q')
def test_uses_topic_channel_artist_for_joint_album(self):
info = {
'album': 'Watch the Throne',
'artists': ['JAY-Z', 'Kanye West'],
'channel': 'JAY-Z & Kanye West - Topic',
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'JAY-Z & Kanye West')
def test_uses_topic_uploader_and_strips_suffix_for_compilation(self):
info = {
'album': 'Compilation',
'artist': 'Track Artist',
'channel': 'Regular Channel',
'uploader': 'Various Artists - Topic',
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'Various Artists')
def test_regular_channel_falls_back_to_main_artist(self):
info = {
'album': 'Album',
'artist': 'Track Artist',
'channel': 'Label Channel',
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'Track Artist')
def test_preserves_explicit_various_artists(self):
info = {
'album': 'Revenge of the Dreamers III',
'artist': 'J. Cole',
'album_artist': 'Various Artists',
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'Various Artists')
def test_preserves_existing_album_artists_list(self):
info = {
'album': 'Album',
'artist': 'Track Artist',
'album_artists': ['Album Artist'],
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artists'], ['Album Artist'])
self.assertNotIn('album_artist', result)
def test_uses_first_artist_when_artist_list_has_multiple_entries(self):
info = {'album': 'Album', 'artists': ['Main Artist', 'Featured Artist']}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'Main Artist')
def test_does_not_fill_without_album(self):
info = {'artist': 'Standalone Artist'}
_, result = self.postprocessor.run(info)
self.assertNotIn('album_artist', result)
self.assertNotIn('album_artists', result)
def test_does_not_fill_without_artist(self):
info = {'album': 'Instrumental Album'}
_, result = self.postprocessor.run(info)
self.assertNotIn('album_artist', result)
self.assertNotIn('album_artists', result)
class AlbumArtistRegistrationTests(unittest.TestCase):
def test_audio_download_registers_pre_process_postprocessor(self):
download = _make_test_download()
download.info.download_type = 'audio'
fake_ydl = MagicMock()
with patch('ytdl._ConfinedYoutubeDL', return_value=fake_ydl):
result = download._make_youtube_dl({'quiet': True})
self.assertIs(result, fake_ydl)
postprocessor, = fake_ydl.add_post_processor.call_args.args
self.assertIsInstance(postprocessor, _AlbumArtistPostProcessor)
self.assertEqual(fake_ydl.add_post_processor.call_args.kwargs, {'when': 'pre_process'})
def test_video_download_does_not_register_postprocessor(self):
download = _make_test_download()
fake_ydl = MagicMock()
with patch('ytdl._ConfinedYoutubeDL', return_value=fake_ydl):
download._make_youtube_dl({'quiet': True})
fake_ydl.add_post_processor.assert_not_called()
class SanitizePathComponentTests(unittest.TestCase):
@@ -61,6 +223,24 @@ class SanitizePathComponentTests(unittest.TestCase):
self.assertIs(_sanitize_path_component(None), None)
self.assertEqual(_sanitize_path_component(42), 42)
def test_strips_path_separators_and_traversal(self):
result = _sanitize_path_component('../../../../etc/x')
self.assertNotIn('..', result)
self.assertNotIn('/', result)
self.assertNotIn('\\', result)
def test_strips_leading_absolute_path_separator(self):
result = _sanitize_path_component('/tmp/x')
self.assertFalse(result.startswith('/'))
self.assertFalse(result.startswith('\\'))
self.assertEqual(result, '_tmp_x')
def test_collapses_slashes_in_legitimate_titles(self):
self.assertEqual(_sanitize_path_component('AC/DC'), 'AC_DC')
def test_empty_after_strip_becomes_underscore(self):
self.assertEqual(_sanitize_path_component(' '), '_')
@unittest.skipUnless(_has_real_ytdlp, "requires real yt-dlp")
class ResolveOuttmplFieldsTests(unittest.TestCase):
@@ -125,6 +305,69 @@ class ResolveOuttmplFieldsTests(unittest.TestCase):
)
self.assertEqual(result, "5 - %(title)s.%(ext)s")
def test_malicious_playlist_title_cannot_escape_via_template(self):
malicious_title = '/tmp/METUBE_ARBITRARY_WRITE_POC'
entry = {
'playlist_title': malicious_title,
'playlist_index': '1',
'title': 'video',
'ext': 'mp4',
}
sanitized = {k: _sanitize_path_component(v) for k, v in entry.items()}
template = '%(playlist_title)s/%(title)s.%(ext)s'
result = _resolve_outtmpl_fields(template, sanitized, ('playlist',))
marker = result.find('%(')
literal_prefix = result[:marker] if marker != -1 else result
self.assertNotIn('..', literal_prefix)
self.assertFalse(literal_prefix.startswith('/'))
self.assertFalse(literal_prefix.startswith('\\'))
class ConfinedYoutubeDLTests(unittest.TestCase):
"""The chokepoint: ``_ConfinedYoutubeDL.prepare_filename`` validates the
*resolved* output path (after yt-dlp expands the template) and refuses any
write outside the allowed roots. This is the single guard for the download-
directory invariant across the main file, split-chapter files, thumbnails,
subtitles, etc. the ``..`` only exists post-expansion, so it is caught here
rather than by any ingress string check.
"""
def setUp(self):
self.base = os.path.realpath(tempfile.mkdtemp())
def _prepared_path(self, resolved, roots=None):
ydl = ytdl._ConfinedYoutubeDL.__new__(ytdl._ConfinedYoutubeDL)
ydl._allowed_roots = [self.base] if roots is None else roots
with patch.object(
ytdl.yt_dlp.YoutubeDL, "prepare_filename", return_value=resolved
):
return ydl.prepare_filename({})
def test_chapter_traversal_via_metadata_is_blocked(self):
# e.g. chapter_template '%(section_title)s/%(section_title)s/pwned.%(ext)s'
# with a chapter titled '..' expands to '../../pwned.mp4'.
escaping = os.path.join(self.base, "..", "..", "pwned.mp4")
with self.assertRaises(ytdl.yt_dlp.utils.DownloadError):
self._prepared_path(escaping)
def test_absolute_output_path_is_blocked(self):
with self.assertRaises(ytdl.yt_dlp.utils.DownloadError):
self._prepared_path("/etc/cron.d/evil")
def test_path_inside_download_dir_is_allowed(self):
ok = os.path.join(self.base, "Playlist", "video.mp4")
self.assertEqual(self._prepared_path(ok), ok)
def test_sibling_prefix_directory_is_blocked(self):
# base '/x/downloads' must not be escapable to '/x/downloads-secret'.
sibling = self.base + "-secret"
with self.assertRaises(ytdl.yt_dlp.utils.DownloadError):
self._prepared_path(os.path.join(sibling, "video.mp4"))
def test_empty_and_stdout_targets_pass_through(self):
self.assertEqual(self._prepared_path(""), "")
self.assertEqual(self._prepared_path("-"), "-")
class SanitizeEntryForPickleTests(unittest.TestCase):
def test_nested(self):
@@ -217,11 +460,16 @@ class ProgressThrottleTests(unittest.TestCase):
class CancelProcessGroupTests(unittest.TestCase):
def test_cancel_kills_group_when_child_is_group_leader(self):
# cancel() now sends SIGINT first (so yt-dlp/ffmpeg can finalize the
# partial file) and schedules a SIGKILL escalation via the event loop
# after ytdl._CANCEL_GRACE_SECONDS, instead of SIGKILLing immediately.
def test_cancel_sends_sigint_to_group_and_schedules_sigkill_escalation(self):
# Child successfully ran os.setpgrp(): its pgid equals its own pid.
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321)
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
dl.loop = MagicMock()
with patch.object(Download, "running", return_value=True), \
patch("ytdl.os.getpgid", return_value=4321) as mock_getpgid, \
@@ -229,38 +477,109 @@ class CancelProcessGroupTests(unittest.TestCase):
dl.cancel()
mock_getpgid.assert_called_once_with(4321)
mock_killpg.assert_called_once_with(4321, signal.SIGKILL)
mock_killpg.assert_called_once_with(4321, signal.SIGINT)
dl.loop.call_later.assert_called_once_with(ytdl._CANCEL_GRACE_SECONDS, dl._kill_if_alive)
self.assertTrue(dl.canceled)
def test_cancel_does_not_killpg_parent_group_kills_child_only(self):
def test_cancel_does_not_killpg_parent_group_signals_child_only(self):
# Child has NOT become its own group leader yet (pgid != pid, e.g. it is
# still in the server's process group). killpg must NOT be called — that
# would SIGKILL the whole server — and we fall back to proc.kill().
# would signal the whole server — and we fall back to os.kill(pid, SIGINT).
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321, kill=MagicMock())
dl.proc = types.SimpleNamespace(pid=4321)
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
dl.loop = MagicMock()
with patch.object(Download, "running", return_value=True), \
patch("ytdl.os.getpgid", return_value=999), \
patch("ytdl.os.killpg") as mock_killpg:
patch("ytdl.os.killpg") as mock_killpg, \
patch("ytdl.os.kill") as mock_kill:
dl.cancel()
mock_killpg.assert_not_called()
dl.proc.kill.assert_called_once()
mock_kill.assert_called_once_with(4321, signal.SIGINT)
dl.loop.call_later.assert_called_once_with(ytdl._CANCEL_GRACE_SECONDS, dl._kill_if_alive)
self.assertTrue(dl.canceled)
def test_cancel_falls_back_to_proc_kill_when_getpgid_unavailable(self):
def test_cancel_falls_back_to_pid_signal_when_getpgid_unavailable(self):
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321)
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
dl.loop = MagicMock()
with patch.object(Download, "running", return_value=True), \
patch("ytdl.os.getpgid", side_effect=OSError("no such process")), \
patch("ytdl.os.kill") as mock_kill:
dl.cancel()
mock_kill.assert_called_once_with(4321, signal.SIGINT)
dl.loop.call_later.assert_called_once_with(ytdl._CANCEL_GRACE_SECONDS, dl._kill_if_alive)
self.assertTrue(dl.canceled)
def test_cancel_kills_immediately_when_signal_delivery_fails(self):
# Neither killpg nor os.kill succeed (process already gone): cancel()
# must fall through to an immediate SIGKILL attempt instead of
# scheduling a pointless escalation.
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321, kill=MagicMock())
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
dl.loop = MagicMock()
with patch.object(Download, "running", return_value=True), \
patch("ytdl.os.getpgid", side_effect=OSError("no such process")):
patch("ytdl.os.getpgid", side_effect=OSError("no such process")), \
patch("ytdl.os.kill", side_effect=ProcessLookupError()):
dl.cancel()
dl.loop.call_later.assert_not_called()
dl.proc.kill.assert_called_once()
self.assertTrue(dl.canceled)
def test_cancel_schedules_escalation_even_without_running_loop(self):
# dl.loop is None (e.g. cancel() called before start()'s run_in_executor
# set it up): _kill_if_alive() must run synchronously instead of being
# scheduled, since there's no loop to schedule it on.
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321)
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
self.assertIsNone(dl.loop)
with patch.object(Download, "running", side_effect=[True, True]), \
patch("ytdl.os.getpgid", return_value=4321), \
patch("ytdl.os.killpg") as mock_killpg:
dl.cancel()
# First SIGINT, then _kill_if_alive() ran inline and sent SIGKILL.
mock_killpg.assert_has_calls([
unittest.mock.call(4321, signal.SIGINT),
unittest.mock.call(4321, signal.SIGKILL),
])
self.assertTrue(dl.canceled)
class KillIfAliveTests(unittest.TestCase):
def test_kill_if_alive_sigkills_running_process(self):
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321)
with patch.object(Download, "running", return_value=True), \
patch("ytdl.os.getpgid", return_value=4321), \
patch("ytdl.os.killpg") as mock_killpg:
dl._kill_if_alive()
mock_killpg.assert_called_once_with(4321, signal.SIGKILL)
def test_kill_if_alive_noop_when_process_already_exited(self):
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321)
with patch.object(Download, "running", return_value=False), \
patch("ytdl.os.killpg") as mock_killpg, \
patch("ytdl.os.kill") as mock_kill:
dl._kill_if_alive()
mock_killpg.assert_not_called()
mock_kill.assert_not_called()
class ConvertSrtToTxtTests(unittest.TestCase):
def test_basic_conversion(self):
+166
View File
@@ -0,0 +1,166 @@
"""Lightweight SSRF guard for user-submitted URLs.
MeTube hands user-submitted URLs to yt-dlp, whose generic extractor will fetch
any ``http(s)`` URL. Without a guard, an attacker can make the server fetch
internal endpoints (cloud metadata services, loopback, RFC1918 hosts, etc.) and
have the response saved to the download directory and served back.
This module provides two layers:
* ``validate_url`` a cheap validator applied at every URL ingress.
* ``install_socket_guard`` a connect-time ``getaddrinfo`` guard installed in
the download subprocess, which re-validates every resolved address and so
covers redirects, DNS rebinding, and media URLs yt-dlp derives from remote
metadata for any backend that resolves through Python's socket module.
Known limitations network isolation (e.g. Docker) remains the backstop for
all of these:
* The socket guard is installed only in the download subprocess. Metadata
extraction (``ytdl.DownloadQueue.__extract_info``) runs in the main process,
where installing a process-wide guard would reject the server's own bind on
``HOST=0.0.0.0``. So extraction which also follows redirects is covered
only by ``validate_url`` at ingress, not at connect time; a redirect from an
allowed host to an internal one during extraction is not blocked (a lower-
impact, blind SSRF, since the extraction response is not written to disk).
* Native resolvers (curl_cffi/libcurl via ``--impersonate``) resolve outside
Python's socket module and bypass the connect-time guard entirely.
"""
import ipaddress
import logging
import socket
from urllib.parse import urlsplit
log = logging.getLogger('url_guard')
_ALLOWED_SCHEMES = ('http', 'https')
# Hostnames that must be blocked without needing a lookup. ``localhost`` and any
# subdomain of it are conventionally loopback, and the GCP metadata name is a
# well-known SSRF target that may resolve via a resolver we don't control.
_BLOCKED_HOSTNAMES = ('localhost', 'metadata.google.internal')
def _hostname_is_blocked(hostname: str) -> bool:
host = hostname.rstrip('.').lower()
for blocked in _BLOCKED_HOSTNAMES:
if host == blocked or host.endswith('.' + blocked):
return True
return False
def _normalise_ip(addr: str):
"""Parse *addr*, unwrapping IPv4-mapped IPv6 (e.g. ``::ffff:169.254.169.254``)
so the embedded IPv4 address is judged on its own merits. Returns ``None``
when *addr* is not a valid IP literal."""
try:
ip = ipaddress.ip_address(addr)
except ValueError:
return None
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
ip = ip.ipv4_mapped
return ip
def _address_is_global(addr: str) -> bool:
ip = _normalise_ip(addr)
return ip is not None and ip.is_global
def _address_allowed_at_connect(addr: str) -> bool:
"""True if *addr* may be connected to at download time.
Permits global addresses and loopback loopback so that locally-configured
proxies (e.g. ``proxy: http://127.0.0.1:9050``) keep working. Blocks the SSRF
targets that matter: link-local (cloud metadata at 169.254.169.254), private
(RFC1918), unique-local and every other non-global, non-loopback range.
"""
ip = _normalise_ip(addr)
return ip is not None and (ip.is_global or ip.is_loopback)
# Captured at import so re-installing the guard never wraps the wrapper.
_real_getaddrinfo = socket.getaddrinfo
def _guarded_getaddrinfo(host, *args, **kwargs):
results = _real_getaddrinfo(host, *args, **kwargs)
allowed = [r for r in results if _address_allowed_at_connect(r[4][0])]
if not allowed:
raise socket.gaierror(f'Refusing to connect to non-global address for host {host!r}')
return allowed
def install_socket_guard(allow_private: bool = False) -> None:
"""Enforce the no-internal-hosts policy at actual connection time.
``validate_url`` only checks the *submitted* URL string; yt-dlp then follows
HTTP redirects and resolves media URLs from remote metadata without
re-validating them. Installing this in the download subprocess re-checks
every resolved address at connect time, covering redirects and DNS rebinding
for any networking backend that resolves through Python's socket module
(urllib, requests). Native resolvers notably curl_cffi/libcurl used by
``--impersonate`` bypass this and rely on network isolation as the backstop.
When *allow_private* is set (``ALLOW_PRIVATE_ADDRESSES``), the guard is not
installed at all, so proxy/VPN setups that route through private or Fake-IP
ranges keep working.
"""
if allow_private:
return
socket.getaddrinfo = _guarded_getaddrinfo
def validate_url(url: str, allow_private: bool = False) -> str | None:
"""Return an error message if the URL is disallowed, else ``None``.
Inputs without a ``://`` scheme separator (bare video IDs, ``ytsearch:``
and other yt-dlp search/extractor prefixes) are allowed unchanged so that
non-URL entries keep working.
When *allow_private* is set (``ALLOW_PRIVATE_ADDRESSES``), the internal-host
and internal-address checks are skipped so that trusted proxy/VPN setups
e.g. Fake-IP clients that resolve YouTube to ``198.18.0.0/15`` can be used.
Scheme validation (http/https only) still applies.
"""
if not isinstance(url, str):
return 'Invalid URL'
candidate = url.strip()
if '://' not in candidate:
# Not an absolute URL: bare video IDs, ytsearch: prefixes, etc.
return None
parts = urlsplit(candidate)
scheme = parts.scheme.lower()
if scheme not in _ALLOWED_SCHEMES:
return f'URL scheme "{parts.scheme}" is not allowed (only http and https)'
hostname = parts.hostname
if not hostname:
return 'URL is missing a host'
if allow_private:
# Environment is explicitly trusted: skip the SSRF address checks.
return None
if _hostname_is_blocked(hostname):
return f'Refusing to fetch internal host "{hostname}"'
try:
addrinfo = socket.getaddrinfo(hostname, parts.port, proto=socket.IPPROTO_TCP)
except socket.gaierror:
# Fail closed: a host we cannot resolve is a host we cannot verify as
# non-internal, so refuse it rather than letting the download proceed
# to a target that may resolve differently at fetch time.
return f'Could not resolve host "{hostname}"'
except (UnicodeError, ValueError):
return f'Invalid host "{hostname}"'
for family, _type, _proto, _canonname, sockaddr in addrinfo:
addr = sockaddr[0]
if not _address_is_global(addr):
return f'Refusing to fetch internal address "{addr}" for host "{hostname}"'
return None
+163 -25
View File
@@ -19,12 +19,14 @@ import types
from typing import Any, Optional
import yt_dlp.networking.impersonate
from yt_dlp.postprocessor.common import PostProcessor
from yt_dlp.utils import STR_FORMAT_RE_TMPL, STR_FORMAT_TYPES
import bg_tasks
from dl_formats import get_format, get_opts, AUDIO_FORMATS, merge_ytdl_option_layers
from datetime import datetime
from state_store import AtomicJsonStore, from_json_compatible, read_legacy_shelf, to_json_compatible
from subscriptions import _entry_id
from url_guard import validate_url, install_socket_guard
log = logging.getLogger('ytdl')
@@ -45,6 +47,10 @@ _MP_CTX = (
else multiprocessing.get_context()
)
# Grace period between SIGINT (lets yt-dlp/ffmpeg finalize the partial file,
# e.g. when cancelling a livestream) and SIGKILL escalation.
_CANCEL_GRACE_SECONDS = 15
_LIVE_CHECK_INTERVAL = 60
_LIVE_MAX_CHECK_INTERVAL = 3600
# Consecutive probe failures (network blips, rate limits, transient extractor
@@ -52,6 +58,52 @@ _LIVE_MAX_CHECK_INTERVAL = 3600
_LIVE_PROBE_MAX_FAILURES = 5
class _AlbumArtistPostProcessor(PostProcessor):
"""Fill missing album-artist metadata from yt-dlp's album-level signals."""
_TOPIC_SUFFIX = ' - Topic'
@staticmethod
def _has_value(value: Any) -> bool:
if isinstance(value, str):
return bool(value.strip())
if isinstance(value, (list, tuple)):
return any(_AlbumArtistPostProcessor._has_value(item) for item in value)
return value is not None
@staticmethod
def _main_artist(info) -> Optional[str]:
artists = info.get('artists')
candidates = artists if isinstance(artists, list) else [info.get('artist')]
for candidate in candidates:
if not isinstance(candidate, str) or not candidate.strip():
continue
# YouTube Music uses a spaced middle dot between credited artists.
# The first credit is the primary artist for normal albums.
return candidate.split(' · ', 1)[0].strip()
return None
@classmethod
def _topic_artist(cls, info) -> Optional[str]:
for field in ('channel', 'uploader'):
value = info.get(field)
if not isinstance(value, str) or not value.endswith(cls._TOPIC_SUFFIX):
continue
if artist := value[:-len(cls._TOPIC_SUFFIX)].strip():
return artist
return None
def run(self, info):
if not self._has_value(info.get('album')):
return [], info
if self._has_value(info.get('album_artist')) or self._has_value(info.get('album_artists')):
return [], info
if artist := self._topic_artist(info) or self._main_artist(info):
info['album_artist'] = artist
return [], info
def _is_within_directory(real_base: str, real_target: str) -> bool:
"""True if ``real_target`` is inside (or equal to) ``real_base``.
@@ -72,6 +124,7 @@ def _is_within_directory(real_base: str, real_target: str) -> bool:
# sanitised when substituting playlist/channel titles into output templates so
# that downloads do not fail on NTFS-mounted volumes or Windows Docker hosts.
_WINDOWS_INVALID_PATH_CHARS = re.compile(r'[\\:*?"<>|]')
_PATH_SEP_OR_TRAVERSAL = re.compile(r'[\\/]|\.\.')
def _sanitize_path_component(value: Any) -> Any:
@@ -81,11 +134,47 @@ def _sanitize_path_component(value: Any) -> Any:
that numeric format specs (e.g. ``%(playlist_index)02d``) still work.
Only string values are sanitised because Windows-invalid characters are
only a concern for human-readable strings (titles, channel names, etc.)
that may end up as directory names.
that may end up as directory names. Path separators and ``..`` segments
are also collapsed so attacker-controlled playlist/channel titles cannot
escape the download directory via the output template.
"""
if not isinstance(value, str):
return value
return _WINDOWS_INVALID_PATH_CHARS.sub('_', value)
value = _WINDOWS_INVALID_PATH_CHARS.sub('_', value)
value = _PATH_SEP_OR_TRAVERSAL.sub('_', value)
return value.lstrip('.').strip() or '_'
class _ConfinedYoutubeDL(yt_dlp.YoutubeDL):
"""A ``YoutubeDL`` that refuses to emit any output path outside the allowed roots.
This is the single authoritative enforcement of MeTube's download-directory
containment invariant. yt-dlp expands output templates at download time using
metadata that is fully attacker-controlled (``%(title)s``, ``%(uploader)s``,
``%(section_title)s`` from chapter titles, ) and, on POSIX hosts, does *not*
neutralise a ``..`` path component so any template segment resolving to
``..`` next to a literal separator (or an absolute template) can traverse out
of the download directory. Every output path main file, split-chapter files,
thumbnails, subtitles, infojson is produced by ``prepare_filename``, so
validating its result here covers them all, regardless of which template or
metadata field carries the traversal. Checking the resolved path (rather than
the template string) is what makes this robust: the ``..`` only exists after
expansion, so no ingress string check can see it.
"""
def __init__(self, params=None, *, allowed_roots=(), **kwargs):
self._allowed_roots = [os.path.realpath(r) for r in allowed_roots if r]
super().__init__(params=params, **kwargs)
def prepare_filename(self, *args, **kwargs):
filename = super().prepare_filename(*args, **kwargs)
if filename and filename != '-' and self._allowed_roots:
resolved = os.path.realpath(filename)
if not any(_is_within_directory(root, resolved) for root in self._allowed_roots):
raise yt_dlp.utils.DownloadError(
f'Refusing to write outside the download directory: {filename}'
)
return filename
# Regex matching yt-dlp output-template field references, e.g. ``%(title)s``
@@ -468,11 +557,12 @@ class Download:
cls.manager.shutdown()
cls.manager = None
def __init__(self, download_dir, temp_dir, output_template, output_template_chapter, quality, format, ytdl_opts, info):
def __init__(self, download_dir, temp_dir, output_template, output_template_chapter, quality, format, ytdl_opts, info, allow_private=False):
self.download_dir = download_dir
self.temp_dir = temp_dir
self.output_template = output_template
self.output_template_chapter = output_template_chapter
self.allow_private = allow_private
self.info = info
self.format = get_format(
getattr(info, 'download_type', 'video'),
@@ -528,6 +618,15 @@ class Download:
return put_status
def _make_youtube_dl(self, params):
ydl = _ConfinedYoutubeDL(
params=params,
allowed_roots=(self.download_dir, self.temp_dir),
)
if getattr(self.info, 'download_type', '') == 'audio':
ydl.add_post_processor(_AlbumArtistPostProcessor(ydl), when='pre_process')
return ydl
def _download(self):
# Run in our own process group so cancel() can SIGKILL the whole
# group (yt-dlp + any ffmpeg children it spawned for merge/postproc),
@@ -537,6 +636,11 @@ class Download:
os.setpgrp()
except OSError:
pass
# Re-validate every outbound connection at fetch time. validate_url only
# saw the submitted URL string; this catches redirects and DNS rebinding
# to internal hosts (cloud metadata, RFC1918) that it cannot. Skipped when
# ALLOW_PRIVATE_ADDRESSES trusts the environment (e.g. Fake-IP proxies).
install_socket_guard(self.allow_private)
log.info(f"Starting download for: {self.info.title} ({self.info.url})")
try:
debug_logging = logging.getLogger().isEnabledFor(logging.DEBUG)
@@ -605,7 +709,7 @@ class Download:
[(start, end)],
)
ret = yt_dlp.YoutubeDL(params=ytdl_params).download([self.info.url])
ret = self._make_youtube_dl(ytdl_params).download([self.info.url])
self.status_queue.put({'status': 'finished' if ret == 0 else 'error'})
log.info(f"Finished download for: {self.info.title}")
except yt_dlp.utils.YoutubeDLError as exc:
@@ -633,29 +737,50 @@ class Download:
self.status_queue.put(None)
await self.status_task
def _signal_group(self, sig):
"""Send *sig* to the download's process group, falling back to the
process itself. Returns True if a signal was delivered.
Only signal the whole group when the child actually became its own
group leader via os.setpgrp() in _download() that sets its pgid
equal to its own pid. If it hasn't run setpgrp() yet, or setpgrp()
failed, its pgid is still the SERVER's group and killpg would signal
the entire MeTube process (PID 1 in Docker). Fall back to signalling
just the child process by pid in that case.
"""
try:
pgid = os.getpgid(self.proc.pid)
if pgid == self.proc.pid:
os.killpg(pgid, sig)
return True
except (OSError, AttributeError):
pass
try:
if sig == signal.SIGINT:
os.kill(self.proc.pid, sig)
else:
self.proc.kill()
return True
except Exception as e:
log.error(f"Error signalling process for {self.info.title}: {e}")
return False
def _kill_if_alive(self):
if self.running():
log.info(f"Escalating cancel to SIGKILL for: {self.info.title}")
self._signal_group(signal.SIGKILL)
def cancel(self):
log.info(f"Cancelling download: {self.info.title}")
if self.running():
killed_group = False
try:
pgid = os.getpgid(self.proc.pid)
# Only kill the whole group (yt-dlp + any ffmpeg children) when
# the child actually became its own group leader via
# os.setpgrp() in _download() — that sets its pgid equal to its
# own pid. If it hasn't run setpgrp() yet, or setpgrp() failed,
# its pgid is still the SERVER's group and killpg would SIGKILL
# the entire MeTube process (PID 1 in Docker). Fall back to
# killing just the child process by pid in that case.
if pgid == self.proc.pid:
os.killpg(pgid, signal.SIGKILL)
killed_group = True
except (OSError, AttributeError):
pass
if not killed_group:
try:
self.proc.kill()
except Exception as e:
log.error(f"Error killing process for {self.info.title}: {e}")
# SIGINT first so yt-dlp/ffmpeg can finalize the partial file
# (livestream recordings stay playable); SIGKILL after a grace
# period if the process ignores it.
interrupted = self._signal_group(signal.SIGINT)
if interrupted and self.loop is not None:
self.loop.call_later(_CANCEL_GRACE_SECONDS, self._kill_if_alive)
else:
self._kill_if_alive()
self.canceled = True
if self.status_queue is not None:
self.status_queue.put(None)
@@ -1146,6 +1271,11 @@ class DownloadQueue:
return opts
def __extract_info(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
# NOTE: extraction runs in the main process, so the connect-time socket
# guard (installed only in the download subprocess) does not apply here.
# The ingress validate_url check guards the submitted URL, but redirects
# followed during extraction are not re-validated. See url_guard's module
# docstring for why the guard can't be installed process-wide.
debug_logging = logging.getLogger().isEnabledFor(logging.DEBUG)
user_opts = self._build_ytdl_options(ytdl_options_presets, ytdl_options_overrides)
params = {
@@ -1205,7 +1335,7 @@ class DownloadQueue:
if playlist_item_limit > 0:
log.info(f'playlist limit is set. Processing only first {playlist_item_limit} entries')
ytdl_options['playlistend'] = playlist_item_limit
download = Download(dldirectory, self.config.TEMP_DIR, output, output_chapter, dl.quality, dl.format, ytdl_options, dl)
download = Download(dldirectory, self.config.TEMP_DIR, output, output_chapter, dl.quality, dl.format, ytdl_options, dl, allow_private=self.config.ALLOW_PRIVATE_ADDRESSES)
is_upcoming = (
getattr(dl, 'live_status', None) == 'is_upcoming'
or getattr(dl, 'status', None) == 'scheduled'
@@ -1425,6 +1555,14 @@ class DownloadQueue:
return {'status': 'ok'}
else:
already.add(url)
# SSRF guard: reject non-http(s) schemes and hosts resolving to
# internal/loopback/link-local/metadata addresses before yt-dlp fetches
# anything. run_in_executor because validate_url may perform a DNS lookup.
url_error = await asyncio.get_running_loop().run_in_executor(
None, partial(validate_url, url, allow_private=self.config.ALLOW_PRIVATE_ADDRESSES))
if url_error is not None:
log.warning('Rejected URL "%s": %s', url, url_error)
return {'status': 'error', 'msg': url_error}
try:
entry = await asyncio.get_running_loop().run_in_executor(
None,
+8 -8
View File
@@ -33,10 +33,10 @@
"@angular/platform-browser-dynamic": "^22.0.6",
"@angular/service-worker": "^22.0.6",
"@fortawesome/angular-fontawesome": "~4.0.0",
"@fortawesome/fontawesome-svg-core": "^7.3.0",
"@fortawesome/free-brands-svg-icons": "^7.3.0",
"@fortawesome/free-regular-svg-icons": "^7.3.0",
"@fortawesome/free-solid-svg-icons": "^7.3.0",
"@fortawesome/fontawesome-svg-core": "^7.3.1",
"@fortawesome/free-brands-svg-icons": "^7.3.1",
"@fortawesome/free-regular-svg-icons": "^7.3.1",
"@fortawesome/free-solid-svg-icons": "^7.3.1",
"@ng-bootstrap/ng-bootstrap": "^21.0.0",
"@ng-select/ng-select": "^23.2.0",
"@popperjs/core": "^2.11.8",
@@ -49,13 +49,13 @@
},
"devDependencies": {
"@angular-eslint/builder": "22.0.0",
"@angular/build": "^22.0.5",
"@angular/cli": "^22.0.5",
"@angular/build": "^22.0.7",
"@angular/cli": "^22.0.7",
"@angular/compiler-cli": "^22.0.6",
"@angular/localize": "^22.0.6",
"@eslint/js": "^9.39.4",
"@eslint/js": "^9.39.5",
"angular-eslint": "22.0.0",
"eslint": "^9.39.4",
"eslint": "^9.39.5",
"jsdom": "^27.4.0",
"typescript": "~6.0.3",
"typescript-eslint": "8.62.0",
+258 -285
View File
File diff suppressed because it is too large Load Diff
Generated
+10 -10
View File
@@ -106,14 +106,14 @@ wheels = [
[[package]]
name = "anyio"
version = "4.14.1"
version = "4.14.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "idna" },
]
sdist = { url = "https://files.pythonhosted.org/packages/3b/72/5562aabb8dd7181e8e860622a38bea08d17842b99ecd4c91f84ac95251b0/anyio-4.14.1.tar.gz", hash = "sha256:8d648a3544c1a700e3ff78615cd679e4c5c3f149904287e73687b2596963629e", size = 254831, upload-time = "2026-06-24T20:56:06.017Z" }
sdist = { url = "https://files.pythonhosted.org/packages/61/cc/a381afa6efea9f496eff839d4a6a1aed3bfafc7b3ab4b0d1b243a12573dd/anyio-4.14.2.tar.gz", hash = "sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f", size = 260176, upload-time = "2026-07-12T20:29:07.082Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/b0/7b/90df4a0a816d98d6ea26f559d87836d494a2cf1fcf063be67df50a7bcc30/anyio-4.14.1-py3-none-any.whl", hash = "sha256:4e5533c5b8ff0a24f5d7a176cbe6877129cd183893f66b537f8f227d10527d72", size = 124875, upload-time = "2026-06-24T20:56:04.413Z" },
{ url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" },
]
[[package]]
@@ -366,15 +366,15 @@ wheels = [
[[package]]
name = "deno"
version = "2.9.2"
version = "2.9.3"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/dc/3c/bf3a4e2c00f7eefd4ed2fd033a958cab3e92dcc539d3ab80351850cae99c/deno-2.9.2.tar.gz", hash = "sha256:3b77d7689c15fb2c47d5a3927dc8cf70dbd408bc92e20c70da079c6d78980e34", size = 8164, upload-time = "2026-07-08T14:38:54.267Z" }
sdist = { url = "https://files.pythonhosted.org/packages/98/ab/638749d76881f74d100a079414745f0a0fd20bba38e27ab4a3a6495d8264/deno-2.9.3.tar.gz", hash = "sha256:73268cbac7f7c4ff1983e49420a93f3f3f2f2e4e4372436f0942d04d02c9e943", size = 8166, upload-time = "2026-07-15T15:37:50.565Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/27/fe/4a02a256f19b3f31ac17ec37f6467ecf523601ed753e6c9177865dd2bbb2/deno-2.9.2-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:fb4bb5362d02c193f8086668789076f366416980b79277c06ea6cff73866b668", size = 42341183, upload-time = "2026-07-08T14:38:36.384Z" },
{ url = "https://files.pythonhosted.org/packages/5a/3e/235530ac9b9c206ee8c098e2b215f711a71d915d8d7d2be77599c8c1db4b/deno-2.9.2-py3-none-macosx_11_0_arm64.whl", hash = "sha256:f6f74fd8698a8a9a31af14eed5730f1964ab50eaa0129079d11d91fa92496f60", size = 37984508, upload-time = "2026-07-08T14:38:40.006Z" },
{ url = "https://files.pythonhosted.org/packages/7b/27/06a166fca538c06dfae4c45dbb92c2ebf3083341141966a3d649272ea734/deno-2.9.2-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:8b2fd9318d452b7e44c45968916670a0c5452d72796e1af9abfaa7a389e4aa6b", size = 42089854, upload-time = "2026-07-08T14:38:43.841Z" },
{ url = "https://files.pythonhosted.org/packages/09/b4/38338550a5ca0a7bcc39991e3d48e46ce3699413460dff32f383a611d822/deno-2.9.2-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:bf0471db7238e90810bca77a8d2c28646317b5a3fd9e5cdfdf73b251fde11b83", size = 43916921, upload-time = "2026-07-08T14:38:48.057Z" },
{ url = "https://files.pythonhosted.org/packages/a9/3d/c490c27e1f720baf014bb2c18b29d09b11c21d69fadfd69639e5904cdc95/deno-2.9.2-py3-none-win_amd64.whl", hash = "sha256:8e815d66b3e1314d028b2f0a05c75a331dd7025f4c825e12dd1f057207cee1f6", size = 41621982, upload-time = "2026-07-08T14:38:51.928Z" },
{ url = "https://files.pythonhosted.org/packages/07/fa/b4e1e2b3b894ed992c78a6796a43dc775edff895239d83c0e9a10457f4f7/deno-2.9.3-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:1deef9fe97ab28d5d4fea07693075e9ccc62f4ea97a653047a770beb01ecd307", size = 42354026, upload-time = "2026-07-15T15:37:34.977Z" },
{ url = "https://files.pythonhosted.org/packages/3d/cc/4090ef7370005b740efe57ec6ff75a3471393abe9816f1459e73dbbf0429/deno-2.9.3-py3-none-macosx_11_0_arm64.whl", hash = "sha256:78635cd9f0802ce7125aa3ba6feb5775460f55d432611374f16b454ffb8a1308", size = 37997909, upload-time = "2026-07-15T15:37:38.291Z" },
{ url = "https://files.pythonhosted.org/packages/14/07/70b7c965bbca5f3d158859acda0e90c041f1fa98b8c9ac8eff3fafd8b96c/deno-2.9.3-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:7c1c669827a480ef9e2710dd762560811da35dca04a9e90b23c0518587dadb8a", size = 42100873, upload-time = "2026-07-15T15:37:41.539Z" },
{ url = "https://files.pythonhosted.org/packages/7b/f5/0a07cc19a27476011719dd017a98950c7ccc33058c2ffc07019ba3ad9a5e/deno-2.9.3-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:3e724d5c8df13a90f6b50b16cc8df1acfe64c61a7736a774e11d61f3bca3a3b1", size = 43928967, upload-time = "2026-07-15T15:37:45.175Z" },
{ url = "https://files.pythonhosted.org/packages/e1/9a/030847bd4ea6cefbb471cae2b63c9ab29a3ab6c1809484447752bbcbcd52/deno-2.9.3-py3-none-win_amd64.whl", hash = "sha256:e2ebe2b5c1a7ee9daeaf8caf14ebc33fda96b4c2399da259e761fe2f82d22fa4", size = 41630201, upload-time = "2026-07-15T15:37:48.344Z" },
]
[[package]]