Compare commits

...

7 Commits

Author SHA1 Message Date
Alex Shnitman e061a8a5ba feat: ALLOW_PRIVATE_ADDRESSES to opt out of the SSRF checks (closes #1036)
The SSRF guard rejects any address that isn't globally routable, which breaks
proxy/VPN setups that resolve hosts into private or special-use ranges. The
reported case is Fake-IP clients (sing-box, Clash, Mihomo) that map YouTube to
the RFC 2544 benchmarking range 198.18.0.0/15 to tunnel the traffic; MeTube
rejected it with "Refusing to fetch internal address" even though yt-dlp itself
handles it fine.

Add a boolean ALLOW_PRIVATE_ADDRESSES (default false) that, when set, skips both
layers: validate_url returns after scheme validation without the internal-host
checks, and the connect-time socket guard is not installed. Threaded to the
download subprocess via Download so the guard sees it too. Scheme validation
(http/https only) still applies. Documented in the README as a trust-your-network
opt-out that disables SSRF protection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 07:44:38 +03:00
Alex Shnitman 13cb65d931 docs: document the SSRF guard's connect-time coverage limitations
The connect-time getaddrinfo guard added for redirect/rebinding SSRF covers
only the download subprocess: metadata extraction runs in the main process
(where a process-wide guard would reject the server's own HOST=0.0.0.0 bind),
and native curl_cffi/libcurl resolution used by --impersonate bypasses Python's
socket module. Record both in url_guard's docstring and at the extraction site
so the boundary is explicit; network isolation remains the backstop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 07:41:01 +03:00
Alex Shnitman 1b02a99510 fix: re-validate outbound connections at fetch time against internal hosts
validate_url only inspects the submitted URL string. yt-dlp then follows HTTP
redirects and resolves media URLs from remote metadata without re-checking, so
an allowed URL that 302s to http://169.254.169.254/ (cloud metadata) or an
RFC1918 host is still fetched — the guard's own docstring scoped this out.

Install a getaddrinfo guard in the download subprocess that re-validates every
resolved address at actual connect time, covering redirects and DNS rebinding
for any backend resolving through Python's socket module (urllib, requests).
Loopback is permitted so locally-configured proxies keep working; link-local,
RFC1918 and unique-local are blocked. Native resolvers (curl_cffi/libcurl via
--impersonate) bypass this and rely on network isolation as the backstop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 06:55:29 +03:00
Alex Shnitman ebcfe577bc fix: fail closed when an SSRF-guarded host cannot be resolved
validate_url() previously returned None (allow) on socket.gaierror, so a
host that failed to resolve at check time was passed straight to yt-dlp. A
host we cannot resolve is a host we cannot verify as non-internal, and it may
resolve differently when yt-dlp fetches it. Reject it instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 23:45:08 +03:00
Alex Shnitman 3bd2c3e366 fix: enforce download-dir containment at the resolved-path chokepoint
The per-download chapter_template was validated only against literal ".."
in the template string, but yt-dlp expands %(section_title)s (and every
other field) from attacker-controlled metadata at download time. On POSIX
hosts yt-dlp does not neutralise a ".." path component, so a chapter titled
".." turns a guard-passing template like
"%(section_title)s/%(section_title)s/x.%(ext)s" into "../../x.mp4" and writes
outside DOWNLOAD_DIR. The same class of escape applies to any multi-segment
output template (default/playlist/channel) whose fields resolve to "..".

The template string can never see the "..": it only exists after expansion.
So move the check to the one point every output path flows through —
YoutubeDL.prepare_filename — via a _ConfinedYoutubeDL subclass that refuses
any resolved path outside the download/temp roots (fail closed). This covers
the main file, split-chapter files, thumbnails and subtitles in one place.

With the chokepoint authoritative, the scattered ingress string checks
(chapter_template, custom_name_prefix) and the weaker _output_dir_escapes
literal-prefix check are removed. Tests move from the ingress layer to the
chokepoint, exercising the real metadata-resolution vector.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 23:44:43 +03:00
Alex Shnitman 707f700609 ci: update releases in place instead of delete-and-recreate
The same-day rerun path deleted and recreated the release tag within
seconds, which corrupted GitHub's release index (release 2026.07.05 was
hidden from the public list and floated to the top for maintainers).
Replace it with gh release create/edit: force-move the tag in place on
reruns so it matches the rebuilt Docker image, exclude today's tag when
collecting release notes so reruns cover the whole day, and pin the
created tag to the triggering commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 16:44:52 +03:00
Alex Shnitman c519f45908 chore: rework issue and discussion templates around scope policy
Issue forms: trim prerequisite checkboxes to the two that matter, make the
yt-dlp test field explicitly conditional (UI bugs write "UI bug"), lead the
feature form with the scope line and the already-decided list, and retire
the question template in favor of Discussions Q&A.

Discussion forms: rename q-and-a.yml to q-a.yml so it matches the actual
category slug (it never applied), add ideas.yml for the category where
feature requests actually land, drop configuration-help.yml (no matching
category) and the inert config.yml (blank_discussions_enabled is not a
GitHub feature), and remove checkboxes from low-stakes discussion forms.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 13:50:34 +03:00
19 changed files with 433 additions and 368 deletions
-1
View File
@@ -1 +0,0 @@
blank_discussions_enabled: false
@@ -1,65 +0,0 @@
name: ⚙️ Configuration Help
description: Get help with MeTube configuration and setup
title: "[Config]: "
labels: ["configuration", "help"]
assignees: []
body:
- type: checkboxes
id: prerequisites
attributes:
label: Prerequisites
description: Please confirm you have completed these steps before asking for configuration help
options:
- label: I have searched existing discussions and issues for similar configuration problems
required: true
- label: I have read the [configuration section](https://github.com/alexta69/metube#%EF%B8%8F-configuration-via-environment-variables) in the README
required: true
- label: I have checked the [Wiki](https://github.com/alexta69/metube/wiki) for configuration examples
required: true
- type: markdown
attributes:
value: |
## Configuration Resources
Before asking for help, please check these resources:
- **[Configuration Guide](https://github.com/alexta69/metube#%EF%B8%8F-configuration-via-environment-variables)** - All available environment variables
- **[YTDL_OPTIONS Cookbook](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook)** - Common yt-dlp configurations
- **[OUTPUT_TEMPLATE Cookbook](https://github.com/alexta69/metube/wiki/OUTPUT_TEMPLATE-Cookbook)** - Filename template examples
- type: textarea
id: configuration-goal
attributes:
label: What are you trying to configure?
description: Describe what you want to achieve with your MeTube configuration
placeholder: |
What specific behavior are you trying to achieve?
What's not working as expected?
What have you tried so far?
validations:
required: true
- type: textarea
id: additional-context
attributes:
label: Additional Context
description: Your configuration, environment details, errors, and any other helpful information
placeholder: |
Current Configuration:
```yaml
# Your docker-compose.yml or environment variables
```
Environment Details:
- MeTube version: [e.g., latest, specific version]
- Docker image: [e.g., ghcr.io/alexta69/metube:latest]
- Operating System: [e.g., Ubuntu 20.04, Windows 10, macOS 12]
Error Messages or Issues:
[Paste any error messages, logs, or unexpected behavior here]
Other relevant information:
[Screenshots, examples, etc.]
validations:
required: true
+5 -31
View File
@@ -1,30 +1,13 @@
name: 💬 General Discussion
description: Start a general discussion about MeTube
title: "[Discussion]: "
labels: ["discussion"]
assignees: []
body: body:
- type: checkboxes
id: prerequisites
attributes:
label: Prerequisites
description: Please confirm you have completed these steps before starting a discussion
options:
- label: I have searched existing discussions to ensure this topic hasn't been discussed before
required: true
- label: I have read the [README](https://github.com/alexta69/metube#readme) and relevant sections
required: true
- type: markdown - type: markdown
attributes: attributes:
value: | value: |
## Discussion Guidelines ## Discussion Guidelines
This is for general discussions about MeTube. For specific issues, please use: This is for general discussions about MeTube. For specific topics, better homes exist:
- **Bug reports** → Use the Bug Report issue template - **Bug reports** → [open an issue](https://github.com/alexta69/metube/issues/new?template=bug_report.yml)
- **Feature requests** → Use the Feature Request issue template - **Feature requests** → [open an issue](https://github.com/alexta69/metube/issues/new?template=feature_request.yml) or post in [Ideas](https://github.com/alexta69/metube/discussions/categories/ideas)
- **Questions** → Use the Question issue template - **Questions** → post in [Q&A](https://github.com/alexta69/metube/discussions/categories/q-a)
- type: textarea - type: textarea
id: discussion-topic id: discussion-topic
@@ -34,12 +17,3 @@ body:
placeholder: Please provide a clear topic for discussion placeholder: Please provide a clear topic for discussion
validations: validations:
required: true required: true
- type: textarea
id: additional-context
attributes:
label: Additional Context
description: Any other information that might be helpful for the discussion
placeholder: Links, examples, or other relevant information
validations:
required: false
+31
View File
@@ -0,0 +1,31 @@
body:
- type: markdown
attributes:
value: |
## Read this first — MeTube's scope
MeTube's scope is deliberately narrow: **it downloads well and stops once the file is written.**
Ideas that improve the download itself are welcome — usually as community PRs, since the
maintainer is unlikely to implement requests ([details](https://github.com/alexta69/metube#-submitting-feature-requests)).
Post-download file management is out of scope regardless of implementation quality.
**Already decided — please don't re-request:**
- 🔒 Built-in authentication / password → use a reverse proxy ([#931](https://github.com/alexta69/metube/issues/931), [wiki](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations))
- 🔔 Notifications on download completion → available today via the `Exec` postprocessor ([recipe](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook))
- ✏️ Renaming, converting, tagging, or organizing files after download → out of scope ([#495](https://github.com/alexta69/metube/issues/495), [#1027](https://github.com/alexta69/metube/issues/1027))
- 🎚️ Pre-download format/quality/audio-track picker → tracked in [#1032](https://github.com/alexta69/metube/issues/1032)
- 📋 Pre-download playlist item browser → tracked in [#1030](https://github.com/alexta69/metube/issues/1030)
- 🎛️ Config switches to disable individual features → declined ([#976](https://github.com/alexta69/metube/issues/976))
- ⏰ Download scheduler → declined; add items with auto-start off and start them when you like ([#838](https://github.com/alexta69/metube/issues/838))
- type: textarea
id: idea
attributes:
label: Your Idea
description: Describe the idea and what problem it solves
placeholder: |
What would you like to see?
What problem does it solve?
Would you be willing to implement it as a PR?
validations:
required: true
+23
View File
@@ -0,0 +1,23 @@
body:
- type: markdown
attributes:
value: |
## Quick Resources
Your question may already be answered here:
- **[README](https://github.com/alexta69/metube#readme)** — complete setup and configuration guide
- **[Troubleshooting FAQ](https://github.com/alexta69/metube/wiki/Troubleshooting-FAQ)** — common problems and their fixes
- **[YTDL_OPTIONS Cookbook](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook)** — common yt-dlp configurations (notifications, metadata, audio extraction, ...)
- **[OUTPUT_TEMPLATE Cookbook](https://github.com/alexta69/metube/wiki/OUTPUT_TEMPLATE-Cookbook)** — filename template examples
- type: textarea
id: question
attributes:
label: Your Question
description: What would you like to know about MeTube?
placeholder: |
What are you trying to achieve?
What's your current setup (docker-compose / environment variables)?
What have you tried so far, and what happened (errors, logs)?
validations:
required: true
-60
View File
@@ -1,60 +0,0 @@
name: ❓ Q&A
description: Ask a question and get answers from the community
title: "[Q&A]: "
labels: ["q-and-a"]
assignees: []
body:
- type: checkboxes
id: prerequisites
attributes:
label: Prerequisites
description: Please confirm you have completed these steps before asking your question
options:
- label: I have searched existing discussions and issues to see if my question has been answered before
required: true
- label: I have read the [README](https://github.com/alexta69/metube#readme) and relevant sections
required: true
- label: I have checked the [Wiki](https://github.com/alexta69/metube/wiki) for configuration examples
required: true
- type: markdown
attributes:
value: |
## Quick Resources
Before asking your question, please check these resources:
- **[README](https://github.com/alexta69/metube#readme)** - Complete setup and configuration guide
- **[YTDL_OPTIONS Cookbook](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook)** - Common yt-dlp configurations
- **[OUTPUT_TEMPLATE Cookbook](https://github.com/alexta69/metube/wiki/OUTPUT_TEMPLATE-Cookbook)** - Filename template examples
- type: textarea
id: question
attributes:
label: Your Question
description: What would you like to know about MeTube?
placeholder: |
What are you trying to achieve?
What's your current setup?
What have you tried so far?
validations:
required: true
- type: textarea
id: additional-context
attributes:
label: Additional Context
description: Any other information that might be helpful (configuration, errors, screenshots, etc.)
placeholder: |
Configuration:
```yaml
# Your docker-compose.yml or environment variables
```
Error messages or logs:
[Paste any error messages or logs here]
Other relevant information:
[Screenshots, examples, etc.]
validations:
required: false
+12 -13
View File
@@ -9,24 +9,20 @@ body:
id: prerequisites id: prerequisites
attributes: attributes:
label: Prerequisites label: Prerequisites
description: Please confirm you have completed these steps before submitting your bug report description: Please confirm before submitting
options: options:
- label: I have searched existing issues and discussions to ensure this bug hasn't been reported before - label: I have searched existing issues and discussions to ensure this bug hasn't been reported before
required: true required: true
- label: I have read the [troubleshooting section](https://github.com/alexta69/metube#-troubleshooting-and-submitting-issues) in the README - label: If the download itself fails, I have tested the same URL and options with yt-dlp directly (see the note below) — or this bug is not download-related
required: true
- label: I have tested this issue with yt-dlp directly (not just through MeTube UI) as described in the README
required: true
- label: I have checked that this is not a yt-dlp issue (if it is, please report it to [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues) instead)
required: true required: true
- type: markdown - type: markdown
attributes: attributes:
value: | value: |
## Important Notes ## Important Notes
- **MeTube is only a UI for yt-dlp** - issues with authentication, postprocessing, permissions, or other yt-dlp functionality should be reported to the [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues) - **MeTube is only a UI for [yt-dlp](https://github.com/yt-dlp/yt-dlp)** issues with authentication, postprocessing, site support, or other download functionality should be reported to the [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues)
- Before reporting, please test with yt-dlp directly using: `docker exec -ti metube sh` then `cd /downloads` and run yt-dlp commands - To test with yt-dlp directly: `docker exec -ti metube sh`, then `cd /downloads` and run `yt-dlp` with your URL and options
- If yt-dlp works directly but MeTube doesn't, then it's a MeTube issue - If yt-dlp works directly but MeTube doesn't, then it's a MeTube issue — report it here
- type: textarea - type: textarea
id: bug-description id: bug-description
@@ -47,10 +43,13 @@ body:
id: ytdl-test-results id: ytdl-test-results
attributes: attributes:
label: yt-dlp Direct Test Results label: yt-dlp Direct Test Results
description: Results of testing the same URL/configuration directly with yt-dlp (required) description: >-
If the download itself fails: paste the exact yt-dlp command you ran AND its output —
download-failure reports without this will be closed as needs-info.
If this is a UI or app bug that doesn't involve a failing download, just write "UI bug".
placeholder: | placeholder: |
Command used: yt-dlp [your-command-here] Command used: yt-dlp [your-command-here]
Result: [success/error and output] Result: [paste the output here]
validations: validations:
required: true required: true
@@ -64,12 +63,12 @@ body:
- MeTube version: [e.g., latest, specific version] - MeTube version: [e.g., latest, specific version]
- Docker image: [e.g., ghcr.io/alexta69/metube:latest] - Docker image: [e.g., ghcr.io/alexta69/metube:latest]
- Operating System: [e.g., Ubuntu 20.04, Windows 10, macOS 12] - Operating System: [e.g., Ubuntu 20.04, Windows 10, macOS 12]
Configuration: Configuration:
```yaml ```yaml
# Your docker-compose.yml or environment variables # Your docker-compose.yml or environment variables
``` ```
Logs: Logs:
```bash ```bash
docker logs metube docker logs metube
+7 -4
View File
@@ -1,8 +1,11 @@
blank_issues_enabled: false blank_issues_enabled: false
contact_links: contact_links:
- name: MeTube Community Discussions - name: ❓ Questions & Support
url: https://github.com/alexta69/metube/discussions url: https://github.com/alexta69/metube/discussions/categories/q-a
about: Ask questions and discuss MeTube with the community about: Ask usage and configuration questions in Discussions Q&A — issues are for bugs and feature requests
- name: yt-dlp Issues - name: 📖 Troubleshooting FAQ
url: https://github.com/alexta69/metube/wiki/Troubleshooting-FAQ
about: Common problems and their fixes
- name: ⬇️ yt-dlp Issues
url: https://github.com/yt-dlp/yt-dlp/issues url: https://github.com/yt-dlp/yt-dlp/issues
about: Report issues related to video downloading, authentication, or site support about: Report issues related to video downloading, authentication, or site support
+22 -15
View File
@@ -5,28 +5,35 @@ labels: ["enhancement"]
assignees: [] assignees: []
body: body:
- type: markdown
attributes:
value: |
## Read this first — MeTube's scope
MeTube's scope is deliberately narrow: **it downloads well and stops once the file is written.**
Features that improve the download itself are welcome — usually as community PRs, since the
maintainer is unlikely to implement requests ([details](https://github.com/alexta69/metube#-submitting-feature-requests)).
Post-download file management is out of scope regardless of implementation quality.
**Already decided — please don't re-request:**
- 🔒 Built-in authentication / password → use a reverse proxy ([#931](https://github.com/alexta69/metube/issues/931), [wiki](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations))
- 🔔 Notifications on download completion → available today via the `Exec` postprocessor ([recipe](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook))
- ✏️ Renaming, converting, tagging, or organizing files after download → out of scope ([#495](https://github.com/alexta69/metube/issues/495), [#1027](https://github.com/alexta69/metube/issues/1027))
- 🎚️ Pre-download format/quality/audio-track picker → tracked in [#1032](https://github.com/alexta69/metube/issues/1032)
- 📋 Pre-download playlist item browser → tracked in [#1030](https://github.com/alexta69/metube/issues/1030)
- 🎛️ Config switches to disable individual features → declined ([#976](https://github.com/alexta69/metube/issues/976))
- ⏰ Download scheduler → declined; add items with auto-start off and start them when you like ([#838](https://github.com/alexta69/metube/issues/838))
- type: checkboxes - type: checkboxes
id: prerequisites id: prerequisites
attributes: attributes:
label: Prerequisites label: Prerequisites
description: Please confirm you have completed these steps before submitting your feature request description: Please confirm before submitting
options: options:
- label: I have searched existing issues and discussions to ensure this feature hasn't been requested before - label: I have searched existing issues and checked the "Already decided" list above
required: true required: true
- label: I have read the [feature request guidelines](https://github.com/alexta69/metube#-submitting-feature-requests) in the README - label: I have considered if this feature belongs in yt-dlp instead (downloading, processing, site support) — if so, please request it in the [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues)
required: true required: true
- label: I understand that MeTube development relies on community contributions and the maintainer is not likely to implement this feature
required: true
- label: I have considered if this feature should be implemented in yt-dlp instead of MeTube (if so, please report to [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues))
required: true
- type: markdown
attributes:
value: |
## Important Notes
- **MeTube development relies on code contributions by the community** - the project is feature-complete for the maintainer's use cases
- **Consider if this belongs in yt-dlp** - if it's related to video downloading, processing, or site support, it might belong in the [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues) instead
- **Some features may not be accepted** - in an effort to reduce bloat, some PRs may not be accepted
- type: textarea - type: textarea
id: feature-description id: feature-description
-62
View File
@@ -1,62 +0,0 @@
name: ❓ Question
description: Ask a question about MeTube usage, configuration, or general help
title: "[Question]: "
labels: ["question"]
assignees: []
body:
- type: checkboxes
id: prerequisites
attributes:
label: Prerequisites
description: Please confirm you have completed these steps before asking your question
options:
- label: I have searched existing issues and discussions to see if my question has been answered before
required: true
- label: I have read the [README](https://github.com/alexta69/metube#readme) and relevant sections
required: true
- label: I have checked the [Wiki](https://github.com/alexta69/metube/wiki) for configuration examples
required: true
- label: I have read the [troubleshooting section](https://github.com/alexta69/metube#-troubleshooting-and-submitting-issues) if this is a technical issue
required: true
- type: markdown
attributes:
value: |
## Quick Resources
Before asking your question, please check these resources:
- **[README](https://github.com/alexta69/metube#readme)** - Complete setup and configuration guide
- **[YTDL_OPTIONS Cookbook](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook)** - Common yt-dlp configurations
- **[OUTPUT_TEMPLATE Cookbook](https://github.com/alexta69/metube/wiki/OUTPUT_TEMPLATE-Cookbook)** - Filename template examples
- type: textarea
id: question
attributes:
label: Your Question
description: What would you like to know about MeTube?
placeholder: |
What are you trying to achieve?
What's your current setup?
What have you tried so far?
validations:
required: true
- type: textarea
id: additional-context
attributes:
label: Additional Context
description: Any other information that might be helpful (configuration, errors, screenshots, etc.)
placeholder: |
Configuration:
```yaml
# Your docker-compose.yml or environment variables
```
Error messages or logs:
[Paste any error messages or logs here]
Other relevant information:
[Screenshots, examples, etc.]
validations:
required: false
+32 -46
View File
@@ -117,25 +117,27 @@ jobs:
- name: Get current date - name: Get current date
id: date id: date
run: echo "date=$(date +'%Y.%m.%d')" >> $GITHUB_OUTPUT run: echo "date=$(date +'%Y.%m.%d')" >> $GITHUB_OUTPUT
- name: Checkout - name: Checkout
uses: actions/checkout@v7 uses: actions/checkout@v7
with: with:
fetch-depth: 0 fetch-depth: 0
- name: Get commits since last release - name: Get commits since last release
id: commits id: commits
env:
DATE: ${{ steps.date.outputs.date }}
run: | run: |
# Fetch all tags
git fetch --tags git fetch --tags
# Get the last tag (sorted by version, using date format YYYY.MM.DD) # Exclude today's tag: on a same-day rerun the notes must cover the
LAST_TAG=$(git tag -l --sort=-version:refname | grep -E '^[0-9]{4}\.[0-9]{2}\.[0-9]{2}$' | head -n 1) # whole day, not just the commits since the morning release.
LAST_TAG=$(git tag -l --sort=-version:refname | grep -E '^[0-9]{4}\.[0-9]{2}\.[0-9]{2}$' | grep -v "^${DATE}$" | head -n 1)
if [ -z "$LAST_TAG" ]; then if [ -z "$LAST_TAG" ]; then
# No previous release, skip commits for first release
COMMITS="" COMMITS=""
echo "has_commits=false" >> $GITHUB_OUTPUT echo "has_commits=false" >> $GITHUB_OUTPUT
else else
# Get commits since last tag
COMMITS=$(git log ${LAST_TAG}..HEAD --pretty=format:"- %s (%h)" --no-merges) COMMITS=$(git log ${LAST_TAG}..HEAD --pretty=format:"- %s (%h)" --no-merges)
if [ -z "$COMMITS" ]; then if [ -z "$COMMITS" ]; then
echo "has_commits=false" >> $GITHUB_OUTPUT echo "has_commits=false" >> $GITHUB_OUTPUT
@@ -143,18 +145,13 @@ jobs:
echo "has_commits=true" >> $GITHUB_OUTPUT echo "has_commits=true" >> $GITHUB_OUTPUT
fi fi
fi fi
# Escape for use in YAML/multiline output
{ {
echo 'commits<<EOF' echo 'commits<<EOF'
echo "$COMMITS" echo "$COMMITS"
echo EOF echo EOF
} >> $GITHUB_OUTPUT } >> $GITHUB_OUTPUT
# Also output for debugging
echo "Last tag: ${LAST_TAG:-none}"
echo "Commits since last release:"
echo "$COMMITS"
- name: Generate release body - name: Generate release body
id: release_body id: release_body
env: env:
@@ -176,7 +173,7 @@ jobs:
echo '**GitHub Container Registry:**' echo '**GitHub Container Registry:**'
echo "- \`${GHCR_REPO}:latest\`" echo "- \`${GHCR_REPO}:latest\`"
echo "- \`${GHCR_REPO}:${DATE}\`" echo "- \`${GHCR_REPO}:${DATE}\`"
if [ "$HAS_COMMITS" = "true" ] && [ -n "$COMMITS" ]; then if [ "$HAS_COMMITS" = "true" ] && [ -n "$COMMITS" ]; then
echo '' echo ''
echo '## Changes' echo '## Changes'
@@ -184,39 +181,28 @@ jobs:
echo "$COMMITS" echo "$COMMITS"
fi fi
} > release_body.txt } > release_body.txt
{ - name: Create or update GitHub Release (mark as latest)
echo 'body<<EOF'
cat release_body.txt
echo EOF
} >> $GITHUB_OUTPUT
- name: Delete existing release if present
env: env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
TAG_NAME: ${{ steps.date.outputs.date }} TAG_NAME: ${{ steps.date.outputs.date }}
run: | run: |
# Check if release exists and delete it if gh release view "$TAG_NAME" >/dev/null 2>&1; then
if gh release view "$TAG_NAME" &>/dev/null; then echo "Release $TAG_NAME exists; updating."
echo "Release $TAG_NAME already exists, deleting it..." # Force-move the tag in place so it matches the rebuilt Docker
gh release delete "$TAG_NAME" --yes || true # image. Never delete+recreate the tag: that corrupts GitHub's
# release index (broke release 2026.07.05).
gh api -X PATCH "repos/${GITHUB_REPOSITORY}/git/refs/tags/${TAG_NAME}" \
-f sha="$GITHUB_SHA" -F force=true
gh release edit "$TAG_NAME" \
--title "Release $TAG_NAME" \
--notes-file release_body.txt \
--latest
else
echo "Release $TAG_NAME does not exist; creating."
gh release create "$TAG_NAME" \
--target "$GITHUB_SHA" \
--title "Release $TAG_NAME" \
--notes-file release_body.txt \
--latest
fi fi
# Fetch tags to check remote
git fetch --tags
# Check if tag exists (locally or remotely) and delete it
if git rev-parse "$TAG_NAME" &>/dev/null 2>&1 || git ls-remote --tags origin "$TAG_NAME" | grep -q "$TAG_NAME"; then
echo "Tag $TAG_NAME already exists, deleting it..."
git tag -d "$TAG_NAME" 2>/dev/null || true
git push origin ":refs/tags/$TAG_NAME" || true
fi
- name: Create GitHub Release
uses: softprops/action-gh-release@v3
with:
tag_name: ${{ steps.date.outputs.date }}
name: Release ${{ steps.date.outputs.date }}
body_path: release_body.txt
draft: false
prerelease: false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+1
View File
@@ -80,6 +80,7 @@ Certain values can be set via environment variables, using the `-e` parameter on
* __YTDL_OPTIONS_PRESETS__: Named bundles of yt-dlp options, selectable per download in the UI. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options) for format and examples. * __YTDL_OPTIONS_PRESETS__: Named bundles of yt-dlp options, selectable per download in the UI. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options) for format and examples.
* __YTDL_OPTIONS_PRESETS_FILE__: Path to a JSON file containing presets. Monitored and reloaded automatically on changes. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options). * __YTDL_OPTIONS_PRESETS_FILE__: Path to a JSON file containing presets. Monitored and reloaded automatically on changes. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options).
* __ALLOW_YTDL_OPTIONS_OVERRIDES__: Whether to show a free-text field in the UI for per-download yt-dlp option overrides. Defaults to `false`. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options) for details and security considerations. * __ALLOW_YTDL_OPTIONS_OVERRIDES__: Whether to show a free-text field in the UI for per-download yt-dlp option overrides. Defaults to `false`. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options) for details and security considerations.
* __ALLOW_PRIVATE_ADDRESSES__: Whether to allow downloads from private, loopback, link-local and other non-global addresses. Defaults to `false`, which protects against SSRF by refusing URLs that resolve to internal hosts. Set to `true` only in trusted environments — for example when routing traffic through a proxy/VPN client in Fake-IP mode (sing-box, Clash, Mihomo), which resolves hosts to the `198.18.0.0/15` range. Enabling this disables the SSRF protection entirely, so only use it when you control the network.
* __YTDL_NIGHTLY_UPDATE_TIME__: If set, will cause MeTube to use [nightly yt-dlp builds](https://github.com/yt-dlp/yt-dlp-nightly-builds) instead of the stable releases. Set to the time (`HH:MM`, 24-hour) when you want the daily upgrades and MeTube restart to happen. Defaults to empty (disabled). * __YTDL_NIGHTLY_UPDATE_TIME__: If set, will cause MeTube to use [nightly yt-dlp builds](https://github.com/yt-dlp/yt-dlp-nightly-builds) instead of the stable releases. Set to the time (`HH:MM`, 24-hour) when you want the daily upgrades and MeTube restart to happen. Defaults to empty (disabled).
### 🌐 Web Server & URLs ### 🌐 Web Server & URLs
+2 -5
View File
@@ -81,6 +81,7 @@ class Config:
'YTDL_OPTIONS_PRESETS': '{}', 'YTDL_OPTIONS_PRESETS': '{}',
'YTDL_OPTIONS_PRESETS_FILE': '', 'YTDL_OPTIONS_PRESETS_FILE': '',
'ALLOW_YTDL_OPTIONS_OVERRIDES': 'false', 'ALLOW_YTDL_OPTIONS_OVERRIDES': 'false',
'ALLOW_PRIVATE_ADDRESSES': 'false',
'CORS_ALLOWED_ORIGINS': '', 'CORS_ALLOWED_ORIGINS': '',
'ROBOTS_TXT': '', 'ROBOTS_TXT': '',
'HOST': '0.0.0.0', 'HOST': '0.0.0.0',
@@ -96,7 +97,7 @@ class Config:
'YTDL_NIGHTLY_UPDATE_TIME': '', 'YTDL_NIGHTLY_UPDATE_TIME': '',
} }
_BOOLEAN = ('DOWNLOAD_DIRS_INDEXABLE', 'CUSTOM_DIRS', 'CREATE_CUSTOM_DIRS', 'DELETE_FILE_ON_TRASHCAN', 'HTTPS', 'ENABLE_ACCESSLOG', 'ALLOW_YTDL_OPTIONS_OVERRIDES') _BOOLEAN = ('DOWNLOAD_DIRS_INDEXABLE', 'CUSTOM_DIRS', 'CREATE_CUSTOM_DIRS', 'DELETE_FILE_ON_TRASHCAN', 'HTTPS', 'ENABLE_ACCESSLOG', 'ALLOW_YTDL_OPTIONS_OVERRIDES', 'ALLOW_PRIVATE_ADDRESSES')
def __init__(self): def __init__(self):
for k, v in self._DEFAULTS.items(): for k, v in self._DEFAULTS.items():
@@ -717,8 +718,6 @@ def parse_download_options(post: dict) -> dict:
if custom_name_prefix is None: if custom_name_prefix is None:
custom_name_prefix = '' custom_name_prefix = ''
if custom_name_prefix and ('..' in custom_name_prefix or custom_name_prefix.startswith('/') or custom_name_prefix.startswith('\\')):
raise web.HTTPBadRequest(reason='custom_name_prefix must not contain ".." or start with a path separator')
if auto_start is None: if auto_start is None:
auto_start = True auto_start = True
if playlist_item_limit is None: if playlist_item_limit is None:
@@ -743,8 +742,6 @@ def parse_download_options(post: dict) -> dict:
enabled=config.ALLOW_YTDL_OPTIONS_OVERRIDES, enabled=config.ALLOW_YTDL_OPTIONS_OVERRIDES,
) )
if chapter_template and ('..' in chapter_template or chapter_template.startswith('/') or chapter_template.startswith('\\')):
raise web.HTTPBadRequest(reason='chapter_template must not contain ".." or start with a path separator')
if not SUBTITLE_LANGUAGE_RE.fullmatch(subtitle_language): if not SUBTITLE_LANGUAGE_RE.fullmatch(subtitle_language):
raise web.HTTPBadRequest(reason='subtitle_language must match pattern [A-Za-z0-9-] and be at most 35 characters') raise web.HTTPBadRequest(reason='subtitle_language must match pattern [A-Za-z0-9-] and be at most 35 characters')
if subtitle_mode not in VALID_SUBTITLE_MODES: if subtitle_mode not in VALID_SUBTITLE_MODES:
+3 -2
View File
@@ -115,7 +115,7 @@ def extract_flat_playlist(
if not nested_url: if not nested_url:
continue continue
# nested_url comes from remote playlist content; guard it too. # nested_url comes from remote playlist content; guard it too.
if validate_url(nested_url) is not None: if validate_url(nested_url, allow_private=getattr(config, "ALLOW_PRIVATE_ADDRESSES", False)) is not None:
continue continue
nested_info, nested_entries = extract_flat_playlist( nested_info, nested_entries = extract_flat_playlist(
config, config,
@@ -548,7 +548,8 @@ class SubscriptionManager:
return {"status": "error", "msg": "Missing URL"} return {"status": "error", "msg": "Missing URL"}
# SSRF guard: block non-http(s) schemes and internal/metadata hosts # SSRF guard: block non-http(s) schemes and internal/metadata hosts
# before yt-dlp fetches the feed. May do a DNS lookup, so run off-loop. # before yt-dlp fetches the feed. May do a DNS lookup, so run off-loop.
url_error = await asyncio.get_running_loop().run_in_executor(None, validate_url, url) url_error = await asyncio.get_running_loop().run_in_executor(
None, partial(validate_url, url, allow_private=getattr(self.config, "ALLOW_PRIVATE_ADDRESSES", False)))
if url_error is not None: if url_error is not None:
log.warning('Rejected subscription URL "%s": %s', url, url_error) log.warning('Rejected subscription URL "%s": %s', url, url_error)
return {"status": "error", "msg": url_error} return {"status": "error", "msg": url_error}
-19
View File
@@ -138,25 +138,6 @@ async def test_add_invalid_subtitle_language(mock_dqueue):
await main.add(req) await main.add(req)
@pytest.mark.asyncio
async def test_add_custom_name_prefix_path_traversal(mock_dqueue):
req = _json_request(_valid_video_add_body(custom_name_prefix="../evil"))
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
@pytest.mark.asyncio
async def test_add_chapter_template_path_traversal(mock_dqueue):
req = _json_request(
_valid_video_add_body(
split_by_chapters=True,
chapter_template="/etc/passwd%(title)s",
)
)
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_add_invalid_json_body(mock_dqueue): async def test_add_invalid_json_body(mock_dqueue):
req = MagicMock(spec=web.Request) req = MagicMock(spec=web.Request)
+89 -3
View File
@@ -6,7 +6,13 @@ import socket
import unittest import unittest
from unittest import mock from unittest import mock
from url_guard import validate_url import url_guard
from url_guard import (
validate_url,
_address_allowed_at_connect,
_guarded_getaddrinfo,
install_socket_guard,
)
def _addrinfo(*addrs, family=socket.AF_INET): def _addrinfo(*addrs, family=socket.AF_INET):
@@ -93,9 +99,89 @@ class AddressResolutionTests(unittest.TestCase):
# If any resolved address is internal, reject the whole URL. # If any resolved address is internal, reject the whole URL.
self.assertIsNotNone(self._validate_with_addrs("http://mixed/x", "142.250.1.1", "127.0.0.1")) self.assertIsNotNone(self._validate_with_addrs("http://mixed/x", "142.250.1.1", "127.0.0.1"))
def test_resolution_failure_defers_to_ytdlp(self): def test_resolution_failure_is_rejected(self):
# Fail closed: an unresolvable host cannot be verified as non-internal.
with mock.patch("url_guard.socket.getaddrinfo", side_effect=socket.gaierror): with mock.patch("url_guard.socket.getaddrinfo", side_effect=socket.gaierror):
self.assertIsNone(validate_url("http://does-not-resolve.example/x")) self.assertIsNotNone(validate_url("http://does-not-resolve.example/x"))
class ConnectAddressPolicyTests(unittest.TestCase):
"""Connect-time policy: allow global + loopback, block everything else."""
def test_global_allowed(self):
self.assertTrue(_address_allowed_at_connect("142.250.1.1"))
def test_loopback_allowed(self):
# Loopback stays reachable so locally-configured proxies keep working.
self.assertTrue(_address_allowed_at_connect("127.0.0.1"))
self.assertTrue(_address_allowed_at_connect("::1"))
def test_link_local_metadata_blocked(self):
self.assertFalse(_address_allowed_at_connect("169.254.169.254"))
def test_private_blocked(self):
self.assertFalse(_address_allowed_at_connect("10.0.0.5"))
self.assertFalse(_address_allowed_at_connect("192.168.1.10"))
def test_ipv4_mapped_metadata_blocked(self):
self.assertFalse(_address_allowed_at_connect("::ffff:169.254.169.254"))
class GuardedGetaddrinfoTests(unittest.TestCase):
def test_internal_only_raises(self):
with mock.patch("url_guard._real_getaddrinfo", return_value=_addrinfo("169.254.169.254")):
with self.assertRaises(socket.gaierror):
_guarded_getaddrinfo("metadata", 80)
def test_filters_internal_keeps_global(self):
# Split-horizon rebinding: keep the public address, drop the internal one.
with mock.patch("url_guard._real_getaddrinfo", return_value=_addrinfo("142.250.1.1", "10.0.0.1")):
results = _guarded_getaddrinfo("mixed", 80)
self.assertEqual([r[4][0] for r in results], ["142.250.1.1"])
def test_loopback_passes(self):
with mock.patch("url_guard._real_getaddrinfo", return_value=_addrinfo("127.0.0.1")):
results = _guarded_getaddrinfo("localproxy", 9050)
self.assertEqual([r[4][0] for r in results], ["127.0.0.1"])
class AllowPrivateBypassTests(unittest.TestCase):
"""ALLOW_PRIVATE_ADDRESSES: trusted proxy/VPN environments opt out of the
SSRF address checks (e.g. Fake-IP clients that resolve to 198.18.0.0/15)."""
def test_internal_address_allowed_when_bypassed(self):
# Fake-IP benchmarking range that is_global rejects by default.
self.assertIsNone(validate_url("http://www.youtube.com/x", allow_private=True))
def test_private_host_allowed_when_bypassed(self):
# No DNS lookup needed: the bypass returns before resolution.
with mock.patch("url_guard.socket.getaddrinfo") as gai:
self.assertIsNone(validate_url("http://192.168.1.1/x", allow_private=True))
gai.assert_not_called()
def test_scheme_still_enforced_when_bypassed(self):
self.assertIsNotNone(validate_url("file:///etc/passwd", allow_private=True))
def test_socket_guard_not_installed_when_bypassed(self):
original = socket.getaddrinfo
try:
install_socket_guard(allow_private=True)
self.assertIs(socket.getaddrinfo, original)
finally:
socket.getaddrinfo = original
class InstallSocketGuardTests(unittest.TestCase):
def test_install_replaces_and_is_idempotent(self):
original = socket.getaddrinfo
try:
install_socket_guard()
self.assertIs(socket.getaddrinfo, url_guard._guarded_getaddrinfo)
# Re-installing must not wrap the wrapper (real fn captured at import).
install_socket_guard()
self.assertIs(socket.getaddrinfo, url_guard._guarded_getaddrinfo)
finally:
socket.getaddrinfo = original
if __name__ == "__main__": if __name__ == "__main__":
+67 -13
View File
@@ -2,6 +2,7 @@
from __future__ import annotations from __future__ import annotations
import os
import pickle import pickle
import signal import signal
import sys import sys
@@ -31,6 +32,25 @@ class _PostProcessor:
self._downloader = downloader self._downloader = downloader
class _YoutubeDL:
"""Minimal stand-in so ``_ConfinedYoutubeDL`` can subclass it under the shim.
``prepare_filename`` is patched per-test; the real containment logic lives in
the ``_ConfinedYoutubeDL`` override, which is what the tests exercise.
"""
def __init__(self, params=None, **kwargs):
self.params = params or {}
def prepare_filename(self, *args, **kwargs):
return ""
def add_post_processor(self, *args, **kwargs):
pass
fake_utils.DownloadError = type("DownloadError", (Exception,), {})
fake_yt_dlp.YoutubeDL = _YoutubeDL
fake_impersonate.ImpersonateTarget = _ImpersonateTarget fake_impersonate.ImpersonateTarget = _ImpersonateTarget
fake_networking.impersonate = fake_impersonate fake_networking.impersonate = fake_impersonate
fake_postprocessor_common.PostProcessor = _PostProcessor fake_postprocessor_common.PostProcessor = _PostProcessor
@@ -56,15 +76,17 @@ from ytdl import (
_compact_persisted_entry, _compact_persisted_entry,
_convert_srt_to_txt_file, _convert_srt_to_txt_file,
_AlbumArtistPostProcessor, _AlbumArtistPostProcessor,
_output_dir_escapes,
_resolve_outtmpl_fields, _resolve_outtmpl_fields,
_sanitize_entry_for_pickle, _sanitize_entry_for_pickle,
_sanitize_path_component, _sanitize_path_component,
) )
# Detect whether the real yt-dlp is loaded (as opposed to the minimal fake # Detect whether the real yt-dlp is loaded (as opposed to the minimal fake
# shim above). _resolve_outtmpl_fields needs YoutubeDL at runtime. # shim above). _resolve_outtmpl_fields needs YoutubeDL.evaluate_outtmpl at
_has_real_ytdlp = hasattr(sys.modules.get("yt_dlp"), "YoutubeDL") # runtime, which the shim's YoutubeDL stand-in deliberately does not provide.
_has_real_ytdlp = hasattr(
getattr(sys.modules.get("yt_dlp"), "YoutubeDL", None), "evaluate_outtmpl"
)
class AlbumArtistPostProcessorTests(unittest.TestCase): class AlbumArtistPostProcessorTests(unittest.TestCase):
@@ -175,7 +197,7 @@ class AlbumArtistRegistrationTests(unittest.TestCase):
download.info.download_type = 'audio' download.info.download_type = 'audio'
fake_ydl = MagicMock() fake_ydl = MagicMock()
with patch('ytdl.yt_dlp.YoutubeDL', return_value=fake_ydl): with patch('ytdl._ConfinedYoutubeDL', return_value=fake_ydl):
result = download._make_youtube_dl({'quiet': True}) result = download._make_youtube_dl({'quiet': True})
self.assertIs(result, fake_ydl) self.assertIs(result, fake_ydl)
@@ -187,7 +209,7 @@ class AlbumArtistRegistrationTests(unittest.TestCase):
download = _make_test_download() download = _make_test_download()
fake_ydl = MagicMock() fake_ydl = MagicMock()
with patch('ytdl.yt_dlp.YoutubeDL', return_value=fake_ydl): with patch('ytdl._ConfinedYoutubeDL', return_value=fake_ydl):
download._make_youtube_dl({'quiet': True}) download._make_youtube_dl({'quiet': True})
fake_ydl.add_post_processor.assert_not_called() fake_ydl.add_post_processor.assert_not_called()
@@ -301,18 +323,50 @@ class ResolveOuttmplFieldsTests(unittest.TestCase):
self.assertFalse(literal_prefix.startswith('\\')) self.assertFalse(literal_prefix.startswith('\\'))
class OutputDirEscapesTests(unittest.TestCase): class ConfinedYoutubeDLTests(unittest.TestCase):
"""The chokepoint: ``_ConfinedYoutubeDL.prepare_filename`` validates the
*resolved* output path (after yt-dlp expands the template) and refuses any
write outside the allowed roots. This is the single guard for the download-
directory invariant across the main file, split-chapter files, thumbnails,
subtitles, etc. the ``..`` only exists post-expansion, so it is caught here
rather than by any ingress string check.
"""
def setUp(self): def setUp(self):
self.base_dir = tempfile.mkdtemp() self.base = os.path.realpath(tempfile.mkdtemp())
def test_relative_traversal_escapes(self): def _prepared_path(self, resolved, roots=None):
self.assertTrue(_output_dir_escapes(self.base_dir, '../../tmp/x/%(title)s.%(ext)s')) ydl = ytdl._ConfinedYoutubeDL.__new__(ytdl._ConfinedYoutubeDL)
ydl._allowed_roots = [self.base] if roots is None else roots
with patch.object(
ytdl.yt_dlp.YoutubeDL, "prepare_filename", return_value=resolved
):
return ydl.prepare_filename({})
def test_absolute_path_escapes(self): def test_chapter_traversal_via_metadata_is_blocked(self):
self.assertTrue(_output_dir_escapes(self.base_dir, '/tmp/x/%(title)s.%(ext)s')) # e.g. chapter_template '%(section_title)s/%(section_title)s/pwned.%(ext)s'
# with a chapter titled '..' expands to '../../pwned.mp4'.
escaping = os.path.join(self.base, "..", "..", "pwned.mp4")
with self.assertRaises(ytdl.yt_dlp.utils.DownloadError):
self._prepared_path(escaping)
def test_normal_playlist_dir_stays_inside(self): def test_absolute_output_path_is_blocked(self):
self.assertFalse(_output_dir_escapes(self.base_dir, 'Playlist/%(title)s.%(ext)s')) with self.assertRaises(ytdl.yt_dlp.utils.DownloadError):
self._prepared_path("/etc/cron.d/evil")
def test_path_inside_download_dir_is_allowed(self):
ok = os.path.join(self.base, "Playlist", "video.mp4")
self.assertEqual(self._prepared_path(ok), ok)
def test_sibling_prefix_directory_is_blocked(self):
# base '/x/downloads' must not be escapable to '/x/downloads-secret'.
sibling = self.base + "-secret"
with self.assertRaises(ytdl.yt_dlp.utils.DownloadError):
self._prepared_path(os.path.join(sibling, "video.mp4"))
def test_empty_and_stdout_targets_pass_through(self):
self.assertEqual(self._prepared_path(""), "")
self.assertEqual(self._prepared_path("-"), "-")
class SanitizeEntryForPickleTests(unittest.TestCase): class SanitizeEntryForPickleTests(unittest.TestCase):
+89 -12
View File
@@ -5,10 +5,26 @@ any ``http(s)`` URL. Without a guard, an attacker can make the server fetch
internal endpoints (cloud metadata services, loopback, RFC1918 hosts, etc.) and internal endpoints (cloud metadata services, loopback, RFC1918 hosts, etc.) and
have the response saved to the download directory and served back. have the response saved to the download directory and served back.
This module provides a single cheap validator applied at every URL ingress. It This module provides two layers:
intentionally does NOT attempt DNS-rebinding pinning, redirect-chain
re-validation, or validation of every media URL yt-dlp derives from remote * ``validate_url`` a cheap validator applied at every URL ingress.
metadata network isolation (e.g. Docker) remains the backstop for those. * ``install_socket_guard`` a connect-time ``getaddrinfo`` guard installed in
the download subprocess, which re-validates every resolved address and so
covers redirects, DNS rebinding, and media URLs yt-dlp derives from remote
metadata for any backend that resolves through Python's socket module.
Known limitations network isolation (e.g. Docker) remains the backstop for
all of these:
* The socket guard is installed only in the download subprocess. Metadata
extraction (``ytdl.DownloadQueue.__extract_info``) runs in the main process,
where installing a process-wide guard would reject the server's own bind on
``HOST=0.0.0.0``. So extraction which also follows redirects is covered
only by ``validate_url`` at ingress, not at connect time; a redirect from an
allowed host to an internal one during extraction is not blocked (a lower-
impact, blind SSRF, since the extraction response is not written to disk).
* Native resolvers (curl_cffi/libcurl via ``--impersonate``) resolve outside
Python's socket module and bypass the connect-time guard entirely.
""" """
import ipaddress import ipaddress
@@ -34,24 +50,79 @@ def _hostname_is_blocked(hostname: str) -> bool:
return False return False
def _address_is_global(addr: str) -> bool: def _normalise_ip(addr: str):
"""Parse *addr*, unwrapping IPv4-mapped IPv6 (e.g. ``::ffff:169.254.169.254``)
so the embedded IPv4 address is judged on its own merits. Returns ``None``
when *addr* is not a valid IP literal."""
try: try:
ip = ipaddress.ip_address(addr) ip = ipaddress.ip_address(addr)
except ValueError: except ValueError:
return False return None
# Unwrap IPv4-mapped/compatible IPv6 (e.g. ::ffff:169.254.169.254) so the
# embedded IPv4 address is judged on its own merits.
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None: if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
ip = ip.ipv4_mapped ip = ip.ipv4_mapped
return ip.is_global return ip
def validate_url(url: str) -> str | None: def _address_is_global(addr: str) -> bool:
ip = _normalise_ip(addr)
return ip is not None and ip.is_global
def _address_allowed_at_connect(addr: str) -> bool:
"""True if *addr* may be connected to at download time.
Permits global addresses and loopback loopback so that locally-configured
proxies (e.g. ``proxy: http://127.0.0.1:9050``) keep working. Blocks the SSRF
targets that matter: link-local (cloud metadata at 169.254.169.254), private
(RFC1918), unique-local and every other non-global, non-loopback range.
"""
ip = _normalise_ip(addr)
return ip is not None and (ip.is_global or ip.is_loopback)
# Captured at import so re-installing the guard never wraps the wrapper.
_real_getaddrinfo = socket.getaddrinfo
def _guarded_getaddrinfo(host, *args, **kwargs):
results = _real_getaddrinfo(host, *args, **kwargs)
allowed = [r for r in results if _address_allowed_at_connect(r[4][0])]
if not allowed:
raise socket.gaierror(f'Refusing to connect to non-global address for host {host!r}')
return allowed
def install_socket_guard(allow_private: bool = False) -> None:
"""Enforce the no-internal-hosts policy at actual connection time.
``validate_url`` only checks the *submitted* URL string; yt-dlp then follows
HTTP redirects and resolves media URLs from remote metadata without
re-validating them. Installing this in the download subprocess re-checks
every resolved address at connect time, covering redirects and DNS rebinding
for any networking backend that resolves through Python's socket module
(urllib, requests). Native resolvers notably curl_cffi/libcurl used by
``--impersonate`` bypass this and rely on network isolation as the backstop.
When *allow_private* is set (``ALLOW_PRIVATE_ADDRESSES``), the guard is not
installed at all, so proxy/VPN setups that route through private or Fake-IP
ranges keep working.
"""
if allow_private:
return
socket.getaddrinfo = _guarded_getaddrinfo
def validate_url(url: str, allow_private: bool = False) -> str | None:
"""Return an error message if the URL is disallowed, else ``None``. """Return an error message if the URL is disallowed, else ``None``.
Inputs without a ``://`` scheme separator (bare video IDs, ``ytsearch:`` Inputs without a ``://`` scheme separator (bare video IDs, ``ytsearch:``
and other yt-dlp search/extractor prefixes) are allowed unchanged so that and other yt-dlp search/extractor prefixes) are allowed unchanged so that
non-URL entries keep working. non-URL entries keep working.
When *allow_private* is set (``ALLOW_PRIVATE_ADDRESSES``), the internal-host
and internal-address checks are skipped so that trusted proxy/VPN setups
e.g. Fake-IP clients that resolve YouTube to ``198.18.0.0/15`` can be used.
Scheme validation (http/https only) still applies.
""" """
if not isinstance(url, str): if not isinstance(url, str):
return 'Invalid URL' return 'Invalid URL'
@@ -70,14 +141,20 @@ def validate_url(url: str) -> str | None:
if not hostname: if not hostname:
return 'URL is missing a host' return 'URL is missing a host'
if allow_private:
# Environment is explicitly trusted: skip the SSRF address checks.
return None
if _hostname_is_blocked(hostname): if _hostname_is_blocked(hostname):
return f'Refusing to fetch internal host "{hostname}"' return f'Refusing to fetch internal host "{hostname}"'
try: try:
addrinfo = socket.getaddrinfo(hostname, parts.port, proto=socket.IPPROTO_TCP) addrinfo = socket.getaddrinfo(hostname, parts.port, proto=socket.IPPROTO_TCP)
except socket.gaierror: except socket.gaierror:
# Let yt-dlp surface a normal resolution error rather than masking it. # Fail closed: a host we cannot resolve is a host we cannot verify as
return None # non-internal, so refuse it rather than letting the download proceed
# to a target that may resolve differently at fetch time.
return f'Could not resolve host "{hostname}"'
except (UnicodeError, ValueError): except (UnicodeError, ValueError):
return f'Invalid host "{hostname}"' return f'Invalid host "{hostname}"'
+50 -17
View File
@@ -26,7 +26,7 @@ from dl_formats import get_format, get_opts, AUDIO_FORMATS, merge_ytdl_option_la
from datetime import datetime from datetime import datetime
from state_store import AtomicJsonStore, from_json_compatible, read_legacy_shelf, to_json_compatible from state_store import AtomicJsonStore, from_json_compatible, read_legacy_shelf, to_json_compatible
from subscriptions import _entry_id from subscriptions import _entry_id
from url_guard import validate_url from url_guard import validate_url, install_socket_guard
log = logging.getLogger('ytdl') log = logging.getLogger('ytdl')
@@ -145,16 +145,36 @@ def _sanitize_path_component(value: Any) -> Any:
return value.lstrip('.').strip() or '_' return value.lstrip('.').strip() or '_'
def _output_dir_escapes(base_dir: str, output_template: str) -> bool: class _ConfinedYoutubeDL(yt_dlp.YoutubeDL):
"""True when the literal directory prefix of *output_template* resolves outside *base_dir*.""" """A ``YoutubeDL`` that refuses to emit any output path outside the allowed roots.
marker = output_template.find('%(')
literal = output_template if marker == -1 else output_template[:marker] This is the single authoritative enforcement of MeTube's download-directory
dir_prefix = os.path.dirname(literal) containment invariant. yt-dlp expands output templates at download time using
if not dir_prefix: metadata that is fully attacker-controlled (``%(title)s``, ``%(uploader)s``,
return False ``%(section_title)s`` from chapter titles, ) and, on POSIX hosts, does *not*
real_base = os.path.realpath(base_dir) neutralise a ``..`` path component so any template segment resolving to
real_target = os.path.realpath(os.path.join(base_dir, dir_prefix)) ``..`` next to a literal separator (or an absolute template) can traverse out
return not _is_within_directory(real_base, real_target) of the download directory. Every output path main file, split-chapter files,
thumbnails, subtitles, infojson is produced by ``prepare_filename``, so
validating its result here covers them all, regardless of which template or
metadata field carries the traversal. Checking the resolved path (rather than
the template string) is what makes this robust: the ``..`` only exists after
expansion, so no ingress string check can see it.
"""
def __init__(self, params=None, *, allowed_roots=(), **kwargs):
self._allowed_roots = [os.path.realpath(r) for r in allowed_roots if r]
super().__init__(params=params, **kwargs)
def prepare_filename(self, *args, **kwargs):
filename = super().prepare_filename(*args, **kwargs)
if filename and filename != '-' and self._allowed_roots:
resolved = os.path.realpath(filename)
if not any(_is_within_directory(root, resolved) for root in self._allowed_roots):
raise yt_dlp.utils.DownloadError(
f'Refusing to write outside the download directory: {filename}'
)
return filename
# Regex matching yt-dlp output-template field references, e.g. ``%(title)s`` # Regex matching yt-dlp output-template field references, e.g. ``%(title)s``
@@ -537,11 +557,12 @@ class Download:
cls.manager.shutdown() cls.manager.shutdown()
cls.manager = None cls.manager = None
def __init__(self, download_dir, temp_dir, output_template, output_template_chapter, quality, format, ytdl_opts, info): def __init__(self, download_dir, temp_dir, output_template, output_template_chapter, quality, format, ytdl_opts, info, allow_private=False):
self.download_dir = download_dir self.download_dir = download_dir
self.temp_dir = temp_dir self.temp_dir = temp_dir
self.output_template = output_template self.output_template = output_template
self.output_template_chapter = output_template_chapter self.output_template_chapter = output_template_chapter
self.allow_private = allow_private
self.info = info self.info = info
self.format = get_format( self.format = get_format(
getattr(info, 'download_type', 'video'), getattr(info, 'download_type', 'video'),
@@ -598,7 +619,10 @@ class Download:
return put_status return put_status
def _make_youtube_dl(self, params): def _make_youtube_dl(self, params):
ydl = yt_dlp.YoutubeDL(params=params) ydl = _ConfinedYoutubeDL(
params=params,
allowed_roots=(self.download_dir, self.temp_dir),
)
if getattr(self.info, 'download_type', '') == 'audio': if getattr(self.info, 'download_type', '') == 'audio':
ydl.add_post_processor(_AlbumArtistPostProcessor(ydl), when='pre_process') ydl.add_post_processor(_AlbumArtistPostProcessor(ydl), when='pre_process')
return ydl return ydl
@@ -612,6 +636,11 @@ class Download:
os.setpgrp() os.setpgrp()
except OSError: except OSError:
pass pass
# Re-validate every outbound connection at fetch time. validate_url only
# saw the submitted URL string; this catches redirects and DNS rebinding
# to internal hosts (cloud metadata, RFC1918) that it cannot. Skipped when
# ALLOW_PRIVATE_ADDRESSES trusts the environment (e.g. Fake-IP proxies).
install_socket_guard(self.allow_private)
log.info(f"Starting download for: {self.info.title} ({self.info.url})") log.info(f"Starting download for: {self.info.title} ({self.info.url})")
try: try:
debug_logging = logging.getLogger().isEnabledFor(logging.DEBUG) debug_logging = logging.getLogger().isEnabledFor(logging.DEBUG)
@@ -1242,6 +1271,11 @@ class DownloadQueue:
return opts return opts
def __extract_info(self, url, ytdl_options_presets=None, ytdl_options_overrides=None): def __extract_info(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
# NOTE: extraction runs in the main process, so the connect-time socket
# guard (installed only in the download subprocess) does not apply here.
# The ingress validate_url check guards the submitted URL, but redirects
# followed during extraction are not re-validated. See url_guard's module
# docstring for why the guard can't be installed process-wide.
debug_logging = logging.getLogger().isEnabledFor(logging.DEBUG) debug_logging = logging.getLogger().isEnabledFor(logging.DEBUG)
user_opts = self._build_ytdl_options(ytdl_options_presets, ytdl_options_overrides) user_opts = self._build_ytdl_options(ytdl_options_presets, ytdl_options_overrides)
params = { params = {
@@ -1301,9 +1335,7 @@ class DownloadQueue:
if playlist_item_limit > 0: if playlist_item_limit > 0:
log.info(f'playlist limit is set. Processing only first {playlist_item_limit} entries') log.info(f'playlist limit is set. Processing only first {playlist_item_limit} entries')
ytdl_options['playlistend'] = playlist_item_limit ytdl_options['playlistend'] = playlist_item_limit
if _output_dir_escapes(dldirectory, output): download = Download(dldirectory, self.config.TEMP_DIR, output, output_chapter, dl.quality, dl.format, ytdl_options, dl, allow_private=self.config.ALLOW_PRIVATE_ADDRESSES)
return {'status': 'error', 'msg': 'Refusing download: resolved output path escapes the download directory'}
download = Download(dldirectory, self.config.TEMP_DIR, output, output_chapter, dl.quality, dl.format, ytdl_options, dl)
is_upcoming = ( is_upcoming = (
getattr(dl, 'live_status', None) == 'is_upcoming' getattr(dl, 'live_status', None) == 'is_upcoming'
or getattr(dl, 'status', None) == 'scheduled' or getattr(dl, 'status', None) == 'scheduled'
@@ -1526,7 +1558,8 @@ class DownloadQueue:
# SSRF guard: reject non-http(s) schemes and hosts resolving to # SSRF guard: reject non-http(s) schemes and hosts resolving to
# internal/loopback/link-local/metadata addresses before yt-dlp fetches # internal/loopback/link-local/metadata addresses before yt-dlp fetches
# anything. run_in_executor because validate_url may perform a DNS lookup. # anything. run_in_executor because validate_url may perform a DNS lookup.
url_error = await asyncio.get_running_loop().run_in_executor(None, validate_url, url) url_error = await asyncio.get_running_loop().run_in_executor(
None, partial(validate_url, url, allow_private=self.config.ALLOW_PRIVATE_ADDRESSES))
if url_error is not None: if url_error is not None:
log.warning('Rejected URL "%s": %s', url, url_error) log.warning('Rejected URL "%s": %s', url, url_error)
return {'status': 'error', 'msg': url_error} return {'status': 'error', 'msg': url_error}