mirror of
https://github.com/alexta69/metube.git
synced 2026-09-21 13:35:01 +00:00
Compare commits
43 Commits
2026.07.13
...
482381d6b9
| Author | SHA1 | Date | |
|---|---|---|---|
| 482381d6b9 | |||
| 0445f5858b | |||
| 6551f7ad58 | |||
| 06c63ec6e5 | |||
| d66b04ccf5 | |||
| 2744f36b44 | |||
| ff1b73a576 | |||
| 1a09dbd686 | |||
| 08dccd98fb | |||
| 1f20aaee94 | |||
| 8a29f3a084 | |||
| 1839e5484d | |||
| fceac97033 | |||
| a13762aa61 | |||
| 50250f8374 | |||
| 926d392926 | |||
| 7f13784445 | |||
| 4aa20890d4 | |||
| e061a8a5ba | |||
| 13cb65d931 | |||
| 1b02a99510 | |||
| 4e27600329 | |||
| ebcfe577bc | |||
| 3bd2c3e366 | |||
| 1c7261ab59 | |||
| 4cf2b1b0d0 | |||
| 707f700609 | |||
| c519f45908 | |||
| a23d1689e3 | |||
| 4b05022b91 | |||
| f3d670e288 | |||
| 0dc9b0b3d6 | |||
| 96f5ffe34a | |||
| 1eafecd1cc | |||
| 8fe81dea18 | |||
| f054d84108 | |||
| edf101faa0 | |||
| fa02717e12 | |||
| 9ca78be199 | |||
| 8071611a84 | |||
| 220f991fae | |||
| 6d0528783c | |||
| c104e30451 |
@@ -1 +0,0 @@
|
||||
blank_discussions_enabled: false
|
||||
@@ -1,65 +0,0 @@
|
||||
name: ⚙️ Configuration Help
|
||||
description: Get help with MeTube configuration and setup
|
||||
title: "[Config]: "
|
||||
labels: ["configuration", "help"]
|
||||
assignees: []
|
||||
|
||||
body:
|
||||
- type: checkboxes
|
||||
id: prerequisites
|
||||
attributes:
|
||||
label: Prerequisites
|
||||
description: Please confirm you have completed these steps before asking for configuration help
|
||||
options:
|
||||
- label: I have searched existing discussions and issues for similar configuration problems
|
||||
required: true
|
||||
- label: I have read the [configuration section](https://github.com/alexta69/metube#%EF%B8%8F-configuration-via-environment-variables) in the README
|
||||
required: true
|
||||
- label: I have checked the [Wiki](https://github.com/alexta69/metube/wiki) for configuration examples
|
||||
required: true
|
||||
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Configuration Resources
|
||||
|
||||
Before asking for help, please check these resources:
|
||||
- **[Configuration Guide](https://github.com/alexta69/metube#%EF%B8%8F-configuration-via-environment-variables)** - All available environment variables
|
||||
- **[YTDL_OPTIONS Cookbook](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook)** - Common yt-dlp configurations
|
||||
- **[OUTPUT_TEMPLATE Cookbook](https://github.com/alexta69/metube/wiki/OUTPUT_TEMPLATE-Cookbook)** - Filename template examples
|
||||
|
||||
- type: textarea
|
||||
id: configuration-goal
|
||||
attributes:
|
||||
label: What are you trying to configure?
|
||||
description: Describe what you want to achieve with your MeTube configuration
|
||||
placeholder: |
|
||||
What specific behavior are you trying to achieve?
|
||||
What's not working as expected?
|
||||
What have you tried so far?
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: additional-context
|
||||
attributes:
|
||||
label: Additional Context
|
||||
description: Your configuration, environment details, errors, and any other helpful information
|
||||
placeholder: |
|
||||
Current Configuration:
|
||||
```yaml
|
||||
# Your docker-compose.yml or environment variables
|
||||
```
|
||||
|
||||
Environment Details:
|
||||
- MeTube version: [e.g., latest, specific version]
|
||||
- Docker image: [e.g., ghcr.io/alexta69/metube:latest]
|
||||
- Operating System: [e.g., Ubuntu 20.04, Windows 10, macOS 12]
|
||||
|
||||
Error Messages or Issues:
|
||||
[Paste any error messages, logs, or unexpected behavior here]
|
||||
|
||||
Other relevant information:
|
||||
[Screenshots, examples, etc.]
|
||||
validations:
|
||||
required: true
|
||||
@@ -1,30 +1,13 @@
|
||||
name: 💬 General Discussion
|
||||
description: Start a general discussion about MeTube
|
||||
title: "[Discussion]: "
|
||||
labels: ["discussion"]
|
||||
assignees: []
|
||||
|
||||
body:
|
||||
- type: checkboxes
|
||||
id: prerequisites
|
||||
attributes:
|
||||
label: Prerequisites
|
||||
description: Please confirm you have completed these steps before starting a discussion
|
||||
options:
|
||||
- label: I have searched existing discussions to ensure this topic hasn't been discussed before
|
||||
required: true
|
||||
- label: I have read the [README](https://github.com/alexta69/metube#readme) and relevant sections
|
||||
required: true
|
||||
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Discussion Guidelines
|
||||
|
||||
This is for general discussions about MeTube. For specific issues, please use:
|
||||
- **Bug reports** → Use the Bug Report issue template
|
||||
- **Feature requests** → Use the Feature Request issue template
|
||||
- **Questions** → Use the Question issue template
|
||||
|
||||
This is for general discussions about MeTube. For specific topics, better homes exist:
|
||||
- **Bug reports** → [open an issue](https://github.com/alexta69/metube/issues/new?template=bug_report.yml)
|
||||
- **Feature requests** → [open an issue](https://github.com/alexta69/metube/issues/new?template=feature_request.yml) or post in [Ideas](https://github.com/alexta69/metube/discussions/categories/ideas)
|
||||
- **Questions** → post in [Q&A](https://github.com/alexta69/metube/discussions/categories/q-a)
|
||||
|
||||
- type: textarea
|
||||
id: discussion-topic
|
||||
@@ -34,12 +17,3 @@ body:
|
||||
placeholder: Please provide a clear topic for discussion
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: additional-context
|
||||
attributes:
|
||||
label: Additional Context
|
||||
description: Any other information that might be helpful for the discussion
|
||||
placeholder: Links, examples, or other relevant information
|
||||
validations:
|
||||
required: false
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Read this first — MeTube's scope
|
||||
|
||||
MeTube's scope is deliberately narrow: **it downloads well and stops once the file is written.**
|
||||
Ideas that improve the download itself are welcome — usually as community PRs, since the
|
||||
maintainer is unlikely to implement requests ([details](https://github.com/alexta69/metube#-submitting-feature-requests)).
|
||||
Post-download file management is out of scope regardless of implementation quality.
|
||||
|
||||
**Already decided — please don't re-request:**
|
||||
- 🔒 Built-in authentication / password → use a reverse proxy ([#931](https://github.com/alexta69/metube/issues/931), [wiki](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations))
|
||||
- 🔔 Notifications on download completion → available today via the `Exec` postprocessor ([recipe](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook))
|
||||
- ✏️ Renaming, converting, tagging, or organizing files after download → out of scope ([#495](https://github.com/alexta69/metube/issues/495), [#1027](https://github.com/alexta69/metube/issues/1027))
|
||||
- 🎚️ Pre-download format/quality/audio-track picker → tracked in [#1032](https://github.com/alexta69/metube/issues/1032)
|
||||
- 📋 Pre-download playlist item browser → tracked in [#1030](https://github.com/alexta69/metube/issues/1030)
|
||||
- 🎛️ Config switches to disable individual features → declined ([#976](https://github.com/alexta69/metube/issues/976))
|
||||
- ⏰ Download scheduler → declined; add items with auto-start off and start them when you like ([#838](https://github.com/alexta69/metube/issues/838))
|
||||
|
||||
- type: textarea
|
||||
id: idea
|
||||
attributes:
|
||||
label: Your Idea
|
||||
description: Describe the idea and what problem it solves
|
||||
placeholder: |
|
||||
What would you like to see?
|
||||
What problem does it solve?
|
||||
Would you be willing to implement it as a PR?
|
||||
validations:
|
||||
required: true
|
||||
@@ -0,0 +1,23 @@
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Quick Resources
|
||||
|
||||
Your question may already be answered here:
|
||||
- **[README](https://github.com/alexta69/metube#readme)** — complete setup and configuration guide
|
||||
- **[Troubleshooting FAQ](https://github.com/alexta69/metube/wiki/Troubleshooting-FAQ)** — common problems and their fixes
|
||||
- **[YTDL_OPTIONS Cookbook](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook)** — common yt-dlp configurations (notifications, metadata, audio extraction, ...)
|
||||
- **[OUTPUT_TEMPLATE Cookbook](https://github.com/alexta69/metube/wiki/OUTPUT_TEMPLATE-Cookbook)** — filename template examples
|
||||
|
||||
- type: textarea
|
||||
id: question
|
||||
attributes:
|
||||
label: Your Question
|
||||
description: What would you like to know about MeTube?
|
||||
placeholder: |
|
||||
What are you trying to achieve?
|
||||
What's your current setup (docker-compose / environment variables)?
|
||||
What have you tried so far, and what happened (errors, logs)?
|
||||
validations:
|
||||
required: true
|
||||
@@ -1,60 +0,0 @@
|
||||
name: ❓ Q&A
|
||||
description: Ask a question and get answers from the community
|
||||
title: "[Q&A]: "
|
||||
labels: ["q-and-a"]
|
||||
assignees: []
|
||||
|
||||
body:
|
||||
- type: checkboxes
|
||||
id: prerequisites
|
||||
attributes:
|
||||
label: Prerequisites
|
||||
description: Please confirm you have completed these steps before asking your question
|
||||
options:
|
||||
- label: I have searched existing discussions and issues to see if my question has been answered before
|
||||
required: true
|
||||
- label: I have read the [README](https://github.com/alexta69/metube#readme) and relevant sections
|
||||
required: true
|
||||
- label: I have checked the [Wiki](https://github.com/alexta69/metube/wiki) for configuration examples
|
||||
required: true
|
||||
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Quick Resources
|
||||
|
||||
Before asking your question, please check these resources:
|
||||
- **[README](https://github.com/alexta69/metube#readme)** - Complete setup and configuration guide
|
||||
- **[YTDL_OPTIONS Cookbook](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook)** - Common yt-dlp configurations
|
||||
- **[OUTPUT_TEMPLATE Cookbook](https://github.com/alexta69/metube/wiki/OUTPUT_TEMPLATE-Cookbook)** - Filename template examples
|
||||
|
||||
- type: textarea
|
||||
id: question
|
||||
attributes:
|
||||
label: Your Question
|
||||
description: What would you like to know about MeTube?
|
||||
placeholder: |
|
||||
What are you trying to achieve?
|
||||
What's your current setup?
|
||||
What have you tried so far?
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: additional-context
|
||||
attributes:
|
||||
label: Additional Context
|
||||
description: Any other information that might be helpful (configuration, errors, screenshots, etc.)
|
||||
placeholder: |
|
||||
Configuration:
|
||||
```yaml
|
||||
# Your docker-compose.yml or environment variables
|
||||
```
|
||||
|
||||
Error messages or logs:
|
||||
[Paste any error messages or logs here]
|
||||
|
||||
Other relevant information:
|
||||
[Screenshots, examples, etc.]
|
||||
validations:
|
||||
required: false
|
||||
@@ -9,24 +9,20 @@ body:
|
||||
id: prerequisites
|
||||
attributes:
|
||||
label: Prerequisites
|
||||
description: Please confirm you have completed these steps before submitting your bug report
|
||||
description: Please confirm before submitting
|
||||
options:
|
||||
- label: I have searched existing issues and discussions to ensure this bug hasn't been reported before
|
||||
required: true
|
||||
- label: I have read the [troubleshooting section](https://github.com/alexta69/metube#-troubleshooting-and-submitting-issues) in the README
|
||||
required: true
|
||||
- label: I have tested this issue with yt-dlp directly (not just through MeTube UI) as described in the README
|
||||
required: true
|
||||
- label: I have checked that this is not a yt-dlp issue (if it is, please report it to [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues) instead)
|
||||
- label: If the download itself fails, I have tested the same URL and options with yt-dlp directly (see the note below) — or this bug is not download-related
|
||||
required: true
|
||||
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Important Notes
|
||||
- **MeTube is only a UI for yt-dlp** - issues with authentication, postprocessing, permissions, or other yt-dlp functionality should be reported to the [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues)
|
||||
- Before reporting, please test with yt-dlp directly using: `docker exec -ti metube sh` then `cd /downloads` and run yt-dlp commands
|
||||
- If yt-dlp works directly but MeTube doesn't, then it's a MeTube issue
|
||||
- **MeTube is only a UI for [yt-dlp](https://github.com/yt-dlp/yt-dlp)** — issues with authentication, postprocessing, site support, or other download functionality should be reported to the [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues)
|
||||
- To test with yt-dlp directly: `docker exec -ti metube sh`, then `cd /downloads` and run `yt-dlp` with your URL and options
|
||||
- If yt-dlp works directly but MeTube doesn't, then it's a MeTube issue — report it here
|
||||
|
||||
- type: textarea
|
||||
id: bug-description
|
||||
@@ -47,10 +43,13 @@ body:
|
||||
id: ytdl-test-results
|
||||
attributes:
|
||||
label: yt-dlp Direct Test Results
|
||||
description: Results of testing the same URL/configuration directly with yt-dlp (required)
|
||||
description: >-
|
||||
If the download itself fails: paste the exact yt-dlp command you ran AND its output —
|
||||
download-failure reports without this will be closed as needs-info.
|
||||
If this is a UI or app bug that doesn't involve a failing download, just write "UI bug".
|
||||
placeholder: |
|
||||
Command used: yt-dlp [your-command-here]
|
||||
Result: [success/error and output]
|
||||
Result: [paste the output here]
|
||||
validations:
|
||||
required: true
|
||||
|
||||
@@ -64,12 +63,12 @@ body:
|
||||
- MeTube version: [e.g., latest, specific version]
|
||||
- Docker image: [e.g., ghcr.io/alexta69/metube:latest]
|
||||
- Operating System: [e.g., Ubuntu 20.04, Windows 10, macOS 12]
|
||||
|
||||
|
||||
Configuration:
|
||||
```yaml
|
||||
# Your docker-compose.yml or environment variables
|
||||
```
|
||||
|
||||
|
||||
Logs:
|
||||
```bash
|
||||
docker logs metube
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: MeTube Community Discussions
|
||||
url: https://github.com/alexta69/metube/discussions
|
||||
about: Ask questions and discuss MeTube with the community
|
||||
- name: yt-dlp Issues
|
||||
- name: ❓ Questions & Support
|
||||
url: https://github.com/alexta69/metube/discussions/categories/q-a
|
||||
about: Ask usage and configuration questions in Discussions Q&A — issues are for bugs and feature requests
|
||||
- name: 📖 Troubleshooting FAQ
|
||||
url: https://github.com/alexta69/metube/wiki/Troubleshooting-FAQ
|
||||
about: Common problems and their fixes
|
||||
- name: ⬇️ yt-dlp Issues
|
||||
url: https://github.com/yt-dlp/yt-dlp/issues
|
||||
about: Report issues related to video downloading, authentication, or site support
|
||||
|
||||
@@ -5,28 +5,35 @@ labels: ["enhancement"]
|
||||
assignees: []
|
||||
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Read this first — MeTube's scope
|
||||
|
||||
MeTube's scope is deliberately narrow: **it downloads well and stops once the file is written.**
|
||||
Features that improve the download itself are welcome — usually as community PRs, since the
|
||||
maintainer is unlikely to implement requests ([details](https://github.com/alexta69/metube#-submitting-feature-requests)).
|
||||
Post-download file management is out of scope regardless of implementation quality.
|
||||
|
||||
**Already decided — please don't re-request:**
|
||||
- 🔒 Built-in authentication / password → use a reverse proxy ([#931](https://github.com/alexta69/metube/issues/931), [wiki](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations))
|
||||
- 🔔 Notifications on download completion → available today via the `Exec` postprocessor ([recipe](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook))
|
||||
- ✏️ Renaming, converting, tagging, or organizing files after download → out of scope ([#495](https://github.com/alexta69/metube/issues/495), [#1027](https://github.com/alexta69/metube/issues/1027))
|
||||
- 🎚️ Pre-download format/quality/audio-track picker → tracked in [#1032](https://github.com/alexta69/metube/issues/1032)
|
||||
- 📋 Pre-download playlist item browser → tracked in [#1030](https://github.com/alexta69/metube/issues/1030)
|
||||
- 🎛️ Config switches to disable individual features → declined ([#976](https://github.com/alexta69/metube/issues/976))
|
||||
- ⏰ Download scheduler → declined; add items with auto-start off and start them when you like ([#838](https://github.com/alexta69/metube/issues/838))
|
||||
|
||||
- type: checkboxes
|
||||
id: prerequisites
|
||||
attributes:
|
||||
label: Prerequisites
|
||||
description: Please confirm you have completed these steps before submitting your feature request
|
||||
description: Please confirm before submitting
|
||||
options:
|
||||
- label: I have searched existing issues and discussions to ensure this feature hasn't been requested before
|
||||
- label: I have searched existing issues and checked the "Already decided" list above
|
||||
required: true
|
||||
- label: I have read the [feature request guidelines](https://github.com/alexta69/metube#-submitting-feature-requests) in the README
|
||||
- label: I have considered if this feature belongs in yt-dlp instead (downloading, processing, site support) — if so, please request it in the [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues)
|
||||
required: true
|
||||
- label: I understand that MeTube development relies on community contributions and the maintainer is not likely to implement this feature
|
||||
required: true
|
||||
- label: I have considered if this feature should be implemented in yt-dlp instead of MeTube (if so, please report to [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues))
|
||||
required: true
|
||||
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Important Notes
|
||||
- **MeTube development relies on code contributions by the community** - the project is feature-complete for the maintainer's use cases
|
||||
- **Consider if this belongs in yt-dlp** - if it's related to video downloading, processing, or site support, it might belong in the [yt-dlp repository](https://github.com/yt-dlp/yt-dlp/issues) instead
|
||||
- **Some features may not be accepted** - in an effort to reduce bloat, some PRs may not be accepted
|
||||
|
||||
- type: textarea
|
||||
id: feature-description
|
||||
|
||||
@@ -1,62 +0,0 @@
|
||||
name: ❓ Question
|
||||
description: Ask a question about MeTube usage, configuration, or general help
|
||||
title: "[Question]: "
|
||||
labels: ["question"]
|
||||
assignees: []
|
||||
|
||||
body:
|
||||
- type: checkboxes
|
||||
id: prerequisites
|
||||
attributes:
|
||||
label: Prerequisites
|
||||
description: Please confirm you have completed these steps before asking your question
|
||||
options:
|
||||
- label: I have searched existing issues and discussions to see if my question has been answered before
|
||||
required: true
|
||||
- label: I have read the [README](https://github.com/alexta69/metube#readme) and relevant sections
|
||||
required: true
|
||||
- label: I have checked the [Wiki](https://github.com/alexta69/metube/wiki) for configuration examples
|
||||
required: true
|
||||
- label: I have read the [troubleshooting section](https://github.com/alexta69/metube#-troubleshooting-and-submitting-issues) if this is a technical issue
|
||||
required: true
|
||||
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Quick Resources
|
||||
|
||||
Before asking your question, please check these resources:
|
||||
- **[README](https://github.com/alexta69/metube#readme)** - Complete setup and configuration guide
|
||||
- **[YTDL_OPTIONS Cookbook](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook)** - Common yt-dlp configurations
|
||||
- **[OUTPUT_TEMPLATE Cookbook](https://github.com/alexta69/metube/wiki/OUTPUT_TEMPLATE-Cookbook)** - Filename template examples
|
||||
|
||||
- type: textarea
|
||||
id: question
|
||||
attributes:
|
||||
label: Your Question
|
||||
description: What would you like to know about MeTube?
|
||||
placeholder: |
|
||||
What are you trying to achieve?
|
||||
What's your current setup?
|
||||
What have you tried so far?
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: additional-context
|
||||
attributes:
|
||||
label: Additional Context
|
||||
description: Any other information that might be helpful (configuration, errors, screenshots, etc.)
|
||||
placeholder: |
|
||||
Configuration:
|
||||
```yaml
|
||||
# Your docker-compose.yml or environment variables
|
||||
```
|
||||
|
||||
Error messages or logs:
|
||||
[Paste any error messages or logs here]
|
||||
|
||||
Other relevant information:
|
||||
[Screenshots, examples, etc.]
|
||||
validations:
|
||||
required: false
|
||||
+33
-47
@@ -14,7 +14,7 @@ jobs:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v7
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v6
|
||||
uses: actions/setup-node@v7
|
||||
with:
|
||||
node-version: lts/*
|
||||
- name: Enable pnpm
|
||||
@@ -117,25 +117,27 @@ jobs:
|
||||
- name: Get current date
|
||||
id: date
|
||||
run: echo "date=$(date +'%Y.%m.%d')" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Get commits since last release
|
||||
id: commits
|
||||
env:
|
||||
DATE: ${{ steps.date.outputs.date }}
|
||||
run: |
|
||||
# Fetch all tags
|
||||
git fetch --tags
|
||||
|
||||
# Get the last tag (sorted by version, using date format YYYY.MM.DD)
|
||||
LAST_TAG=$(git tag -l --sort=-version:refname | grep -E '^[0-9]{4}\.[0-9]{2}\.[0-9]{2}$' | head -n 1)
|
||||
|
||||
|
||||
# Exclude today's tag: on a same-day rerun the notes must cover the
|
||||
# whole day, not just the commits since the morning release.
|
||||
LAST_TAG=$(git tag -l --sort=-version:refname | grep -E '^[0-9]{4}\.[0-9]{2}\.[0-9]{2}$' | grep -v "^${DATE}$" | head -n 1)
|
||||
|
||||
if [ -z "$LAST_TAG" ]; then
|
||||
# No previous release, skip commits for first release
|
||||
COMMITS=""
|
||||
echo "has_commits=false" >> $GITHUB_OUTPUT
|
||||
else
|
||||
# Get commits since last tag
|
||||
COMMITS=$(git log ${LAST_TAG}..HEAD --pretty=format:"- %s (%h)" --no-merges)
|
||||
if [ -z "$COMMITS" ]; then
|
||||
echo "has_commits=false" >> $GITHUB_OUTPUT
|
||||
@@ -143,18 +145,13 @@ jobs:
|
||||
echo "has_commits=true" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
fi
|
||||
|
||||
# Escape for use in YAML/multiline output
|
||||
|
||||
{
|
||||
echo 'commits<<EOF'
|
||||
echo "$COMMITS"
|
||||
echo EOF
|
||||
} >> $GITHUB_OUTPUT
|
||||
|
||||
# Also output for debugging
|
||||
echo "Last tag: ${LAST_TAG:-none}"
|
||||
echo "Commits since last release:"
|
||||
echo "$COMMITS"
|
||||
|
||||
- name: Generate release body
|
||||
id: release_body
|
||||
env:
|
||||
@@ -176,7 +173,7 @@ jobs:
|
||||
echo '**GitHub Container Registry:**'
|
||||
echo "- \`${GHCR_REPO}:latest\`"
|
||||
echo "- \`${GHCR_REPO}:${DATE}\`"
|
||||
|
||||
|
||||
if [ "$HAS_COMMITS" = "true" ] && [ -n "$COMMITS" ]; then
|
||||
echo ''
|
||||
echo '## Changes'
|
||||
@@ -184,39 +181,28 @@ jobs:
|
||||
echo "$COMMITS"
|
||||
fi
|
||||
} > release_body.txt
|
||||
|
||||
{
|
||||
echo 'body<<EOF'
|
||||
cat release_body.txt
|
||||
echo EOF
|
||||
} >> $GITHUB_OUTPUT
|
||||
- name: Delete existing release if present
|
||||
|
||||
- name: Create or update GitHub Release (mark as latest)
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
TAG_NAME: ${{ steps.date.outputs.date }}
|
||||
run: |
|
||||
# Check if release exists and delete it
|
||||
if gh release view "$TAG_NAME" &>/dev/null; then
|
||||
echo "Release $TAG_NAME already exists, deleting it..."
|
||||
gh release delete "$TAG_NAME" --yes || true
|
||||
if gh release view "$TAG_NAME" >/dev/null 2>&1; then
|
||||
echo "Release $TAG_NAME exists; updating."
|
||||
# Force-move the tag in place so it matches the rebuilt Docker
|
||||
# image. Never delete+recreate the tag: that corrupts GitHub's
|
||||
# release index (broke release 2026.07.05).
|
||||
gh api -X PATCH "repos/${GITHUB_REPOSITORY}/git/refs/tags/${TAG_NAME}" \
|
||||
-f sha="$GITHUB_SHA" -F force=true
|
||||
gh release edit "$TAG_NAME" \
|
||||
--title "Release $TAG_NAME" \
|
||||
--notes-file release_body.txt \
|
||||
--latest
|
||||
else
|
||||
echo "Release $TAG_NAME does not exist; creating."
|
||||
gh release create "$TAG_NAME" \
|
||||
--target "$GITHUB_SHA" \
|
||||
--title "Release $TAG_NAME" \
|
||||
--notes-file release_body.txt \
|
||||
--latest
|
||||
fi
|
||||
|
||||
# Fetch tags to check remote
|
||||
git fetch --tags
|
||||
|
||||
# Check if tag exists (locally or remotely) and delete it
|
||||
if git rev-parse "$TAG_NAME" &>/dev/null 2>&1 || git ls-remote --tags origin "$TAG_NAME" | grep -q "$TAG_NAME"; then
|
||||
echo "Tag $TAG_NAME already exists, deleting it..."
|
||||
git tag -d "$TAG_NAME" 2>/dev/null || true
|
||||
git push origin ":refs/tags/$TAG_NAME" || true
|
||||
fi
|
||||
- name: Create GitHub Release
|
||||
uses: softprops/action-gh-release@v3
|
||||
with:
|
||||
tag_name: ${{ steps.date.outputs.date }}
|
||||
name: Release ${{ steps.date.outputs.date }}
|
||||
body_path: release_body.txt
|
||||
draft: false
|
||||
prerelease: false
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
name: readme-size
|
||||
|
||||
on:
|
||||
push:
|
||||
paths:
|
||||
- 'README.md'
|
||||
pull_request:
|
||||
paths:
|
||||
- 'README.md'
|
||||
|
||||
jobs:
|
||||
check-size:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v7
|
||||
- name: Check README stays under Docker Hub's 25k character limit
|
||||
run: |
|
||||
size=$(wc -c < README.md)
|
||||
echo "README.md is ${size} bytes (limit 25000)"
|
||||
test "$size" -lt 25000
|
||||
@@ -15,7 +15,7 @@ jobs:
|
||||
token: ${{ secrets.AUTOUPDATE_PAT }}
|
||||
-
|
||||
name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
uses: actions/setup-python@v7
|
||||
with:
|
||||
python-version: '3.13'
|
||||
-
|
||||
|
||||
@@ -1,5 +1,53 @@
|
||||
# Agent Guidelines
|
||||
|
||||
## Project scope — read this before planning a feature
|
||||
|
||||
MeTube's contract is: give it a URL, it runs yt-dlp well, and correct files appear.
|
||||
The maintainer holds a deliberate line on what belongs inside that contract, and PRs
|
||||
on the wrong side of it are declined **regardless of code quality**. Check your plan
|
||||
against this line before writing any code.
|
||||
|
||||
**In scope — improving the write itself:**
|
||||
|
||||
- Features that make the file yt-dlp writes at download time come out more correct,
|
||||
using only data the extractor already provides (e.g. filling a missing album-artist
|
||||
tag from the extractor's own metadata).
|
||||
- Surfacing functionality yt-dlp itself owns and maintains as first-class UI options
|
||||
(e.g. a SponsorBlock toggle that just passes yt-dlp postprocessor params).
|
||||
- Download queue, subscriptions, output templates, and UI improvements to the
|
||||
download workflow.
|
||||
|
||||
**Out of scope — managing files after they exist:**
|
||||
|
||||
- Tag editors, metadata dialogs, or any workflow that rewrites files after the
|
||||
download has finished. This holds even for slimmed-down versions.
|
||||
- Lookups against external metadata services (iTunes, Deezer, MusicBrainz, etc.).
|
||||
More broadly: any new dependency on a third-party API, or new network egress from
|
||||
self-hosted instances, beyond what yt-dlp itself performs.
|
||||
- Library organization: moving/renaming existing files into Artist/Album layouts,
|
||||
watch-folder processing, and similar media-manager features. Dedicated tools
|
||||
(beets, MusicBrainz Picard, Lidarr) do this properly; the README points users
|
||||
to them.
|
||||
|
||||
**Corollaries that shape borderline PRs:**
|
||||
|
||||
- Site-specific intelligence (parsing playlist-ID prefixes, URL path conventions,
|
||||
and other platform internals) is extractor work and belongs upstream in yt-dlp,
|
||||
not re-implemented here — it silently breaks when the platform changes and
|
||||
MeTube would own the breakage.
|
||||
- Prefer enriching yt-dlp's info dict and letting its existing pipeline
|
||||
(FFmpegMetadata etc.) do the writing, over adding custom per-format tag-writing
|
||||
code to MeTube.
|
||||
- Supplemental processing must never fail a download that otherwise succeeded:
|
||||
warn and continue, don't raise.
|
||||
- Keep feature scope minimal on first submission. A hardcoded sensible default
|
||||
beats a configuration surface; follow-ups can add options when users actually
|
||||
ask. PRs that bundle several "reasonable next steps" invite rejection of the
|
||||
whole.
|
||||
|
||||
If a feature idea fails this test, the accepted alternative is usually a README
|
||||
section documenting how to pair MeTube with the right dedicated tool.
|
||||
|
||||
## README.md size constraint
|
||||
|
||||
The README.md is synced to Docker Hub, which has a **25,000 character limit**.
|
||||
@@ -30,7 +78,41 @@ uv run pytest app/tests/
|
||||
|
||||
All of these run in CI (`.github/workflows/main.yml`) on every push to master and must pass.
|
||||
|
||||
## Code style
|
||||
Gotchas:
|
||||
|
||||
- Backend tests must run **from the repo root**: `main.py` resolves the static-assets
|
||||
path relative to the cwd, and several test modules import `main`. Running from
|
||||
`app/` makes five test modules fail to import.
|
||||
- The frontend must be **built before** running backend tests (same reason — the
|
||||
assets at `ui/dist/metube/browser` must exist). The command order above is
|
||||
load-bearing.
|
||||
- `app/tests/test_ytdl_utils.py` stubs `yt_dlp` at import time. Run standalone,
|
||||
two tests fail with `AttributeError: <module 'yt_dlp'> does not have the
|
||||
attribute 'YoutubeDL'`; under the full suite the real module is imported first
|
||||
and they pass. This is a known quirk, not a bug to fix in the code under test.
|
||||
|
||||
Every non-markdown push to master builds multi-arch Docker images and cuts a dated
|
||||
release the same day. **Master is continuously released** — a PR must be
|
||||
release-ready exactly as merged; there is no stabilization window for follow-up
|
||||
fixes.
|
||||
|
||||
## Commit messages
|
||||
|
||||
A commit that resolves an issue must close it, with a GitHub closing keyword in
|
||||
parentheses at the end of the subject line:
|
||||
|
||||
```
|
||||
fix: stop metadata probes from writing playlist sidecar files (closes #1040)
|
||||
```
|
||||
|
||||
Because master is the default branch and is released on every push, the issue
|
||||
closes at the moment the fix ships, and keeps a permanent link to the commit that
|
||||
fixed it. A bare `(#1040)` is only a reference — and reads as a pull-request
|
||||
number — so it does not count; the keyword is what closes the issue.
|
||||
|
||||
Auto-closing leaves only a commit stub on the issue, which is not an answer to
|
||||
whoever reported it. Post an explanatory comment as well: what the cause was, what
|
||||
changed, and anything the reporter needs to do differently.
|
||||
|
||||
Follow `.editorconfig`:
|
||||
- Python: 4-space indent
|
||||
@@ -56,5 +138,36 @@ ui/src/app/ — Angular standalone components (no NgModules)
|
||||
- Backend configuration lives in the `Config` class in `app/main.py` with env-var defaults in `_DEFAULTS`. New env vars go there.
|
||||
- Real-time communication uses Socket.IO events, not REST polling.
|
||||
- Frontend uses standalone Angular components with `inject()` for DI, RxJS Subjects for state, and `takeUntilDestroyed()` for cleanup.
|
||||
- Frontend components use OnPush change detection: subscribe callbacks must call `cdr.markForCheck()`.
|
||||
- State is persisted as JSON files via `AtomicJsonStore` in `app/state_store.py`.
|
||||
- Persisted state stays compact: the completed queue deliberately drops bulky entry data (see `_compact_persisted_entry` in `app/ytdl.py`). Don't expand what gets persisted without discussion.
|
||||
- Custom yt-dlp postprocessors added to `ytdl_params['postprocessors']` run in **list order** within a stage. When combining postprocessors, mirror the ordering the yt-dlp CLI would produce (e.g. sponsor-segment removal before chapter splitting).
|
||||
- No pre-commit hooks — linting and tests are enforced in CI only.
|
||||
|
||||
## Checklist: adding a per-download option
|
||||
|
||||
New options on the download form (the `split_by_chapters` pattern) need **all** of
|
||||
these pieces — the last three are the ones commonly missed:
|
||||
|
||||
1. `parse_download_options` in `app/main.py`.
|
||||
2. A field on `DownloadInfo` in `app/ytdl.py`.
|
||||
3. A `hasattr` backfill in `DownloadInfo.__setstate__` for old persisted records.
|
||||
4. The safe-deserialization field list in `app/ytdl.py`.
|
||||
5. UI form control + cookie persistence in `ui/src/app/app.ts` / `app.html`, and
|
||||
the payload in `downloads.service.ts` (plus its spec).
|
||||
6. The redownload path in `app.ts`, so retries carry the option.
|
||||
7. If the option makes sense for unattended downloads: threading through
|
||||
`app/subscriptions.py` (`SubscriptionInfo` field, serializer, add/update
|
||||
routes, the enqueue call) — or a note in the PR that it's deliberately
|
||||
direct-downloads-only.
|
||||
|
||||
## Security invariants
|
||||
|
||||
User input and extractor-provided metadata (titles, playlist names, URLs) are
|
||||
untrusted. Use the existing guards instead of hand-rolling:
|
||||
|
||||
- User-submitted URLs go through the SSRF guard (see `test_url_guard.py` for the
|
||||
expected behavior).
|
||||
- Anything that becomes a filesystem path goes through `_is_within_directory` and
|
||||
`_sanitize_path_component` in `app/ytdl.py` — including values that arrive via
|
||||
yt-dlp metadata, which sites can influence.
|
||||
|
||||
@@ -3,14 +3,14 @@
|
||||

|
||||

|
||||
|
||||
MeTube is a self-hosted web UI for `yt-dlp`, for downloading media from YouTube and [dozens of other sites](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md).
|
||||
MeTube is a self-hosted web UI for `yt-dlp`, for downloading media from YouTube and [dozens of other sites](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md). Docker images are multi-arch (amd64/arm64).
|
||||
|
||||
Key capabilities:
|
||||
* Download videos, audio, captions, and thumbnails from a browser UI.
|
||||
* Download playlists and channels, with configurable output and download options.
|
||||
* Subscribe to channels and playlists, periodically check for new items, and queue new uploads automatically.
|
||||
* [Subscribe](https://github.com/alexta69/metube/wiki/Subscriptions) to channels and playlists, periodically check for new items, and queue new uploads automatically.
|
||||
|
||||

|
||||

|
||||
|
||||
## 🐳 Run using Docker
|
||||
|
||||
@@ -18,7 +18,7 @@ Key capabilities:
|
||||
docker run -d -p 8081:8081 -v /path/to/downloads:/downloads ghcr.io/alexta69/metube
|
||||
```
|
||||
|
||||
## 🐳 Run using docker-compose
|
||||
## 🐳 Run using Docker Compose
|
||||
|
||||
```yaml
|
||||
services:
|
||||
@@ -34,11 +34,20 @@ services:
|
||||
|
||||
## ⚙️ Configuration via environment variables
|
||||
|
||||
Certain values can be set via environment variables, using the `-e` parameter on the docker command line, or the `environment:` section in docker-compose.
|
||||
Certain values can be set via environment variables, using the `-e` parameter on the docker command line, or the `environment:` section in Docker Compose.
|
||||
|
||||
### 🏠 Runtime & Permissions
|
||||
|
||||
* __PUID__: User under which MeTube will run. Defaults to `1000` (legacy `UID` also supported).
|
||||
* __PGID__: Group under which MeTube will run. Defaults to `1000` (legacy `GID` also supported).
|
||||
* __UMASK__: Umask value used by MeTube. Defaults to `022`.
|
||||
* __DEFAULT_THEME__: Default theme to use for the UI, can be set to `light`, `dark`, or `auto`. Defaults to `auto`.
|
||||
* __LOGLEVEL__: Log level, can be set to `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, or `NONE`. Defaults to `INFO`.
|
||||
* __ENABLE_ACCESSLOG__: Whether to enable access log. Defaults to `false`.
|
||||
|
||||
### ⬇️ Download Behavior
|
||||
|
||||
* __MAX_CONCURRENT_DOWNLOADS__: Maximum number of simultaneous downloads allowed. For example, if set to `5`, then at most five downloads will run concurrently, and any additional downloads will wait until one of the active downloads completes. Defaults to `3`.
|
||||
* __MAX_CONCURRENT_DOWNLOADS__: Maximum number of simultaneous downloads allowed. For example, if set to `5`, then at most five downloads will run concurrently, and any additional downloads will wait until one of the active downloads completes. Defaults to `3`.
|
||||
* __DELETE_FILE_ON_TRASHCAN__: if `true`, downloaded files are deleted on the server, when they are trashed from the "Completed" section of the UI. Defaults to `false`.
|
||||
* __DEFAULT_OPTION_PLAYLIST_ITEM_LIMIT__: Maximum number of playlist items that can be downloaded. Defaults to `0` (no limit).
|
||||
* __SUBSCRIPTION_DEFAULT_CHECK_INTERVAL__: Default minutes between automatic checks for each subscription. Defaults to `60`.
|
||||
@@ -71,8 +80,11 @@ Certain values can be set via environment variables, using the `-e` parameter on
|
||||
* __YTDL_OPTIONS_PRESETS__: Named bundles of yt-dlp options, selectable per download in the UI. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options) for format and examples.
|
||||
* __YTDL_OPTIONS_PRESETS_FILE__: Path to a JSON file containing presets. Monitored and reloaded automatically on changes. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options).
|
||||
* __ALLOW_YTDL_OPTIONS_OVERRIDES__: Whether to show a free-text field in the UI for per-download yt-dlp option overrides. Defaults to `false`. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options) for details and security considerations.
|
||||
* __ALLOW_PRIVATE_ADDRESSES__: Whether to allow downloads from private, loopback, link-local and other non-global addresses. Defaults to `false`, which protects against SSRF by refusing URLs that resolve to internal hosts. Set to `true` only in trusted environments — for example when routing traffic through a proxy/VPN client in Fake-IP mode (sing-box, Clash, Mihomo), which resolves hosts to the `198.18.0.0/15` range. Enabling this disables the SSRF protection entirely, so only use it when you control the network.
|
||||
* __YTDL_NIGHTLY_UPDATE_TIME__: If set, will cause MeTube to use [nightly yt-dlp builds](https://github.com/yt-dlp/yt-dlp-nightly-builds) instead of the stable releases. Set to the time (`HH:MM`, 24-hour) when you want the daily upgrades and MeTube restart to happen. Defaults to empty (disabled).
|
||||
|
||||
Enabling `writeinfojson` or `writethumbnail` in `YTDL_OPTIONS` also writes a feed-level `.info.json` and thumbnail when you add a playlist or channel. These reuse the template of the items they belong to — `OUTPUT_TEMPLATE_CHANNEL` or `OUTPUT_TEMPLATE_PLAYLIST` — evaluated against the feed itself, so with the defaults they land in the same folder as the videos, named after the feed. Set `allow_playlist_files` to `false` in `YTDL_OPTIONS` to skip them.
|
||||
|
||||
### 🌐 Web Server & URLs
|
||||
|
||||
* __HOST__: The host address the web server will bind to. Defaults to `0.0.0.0` (all interfaces).
|
||||
@@ -83,18 +95,9 @@ Certain values can be set via environment variables, using the `-e` parameter on
|
||||
* __HTTPS__: Use `https` instead of `http` (__CERTFILE__ and __KEYFILE__ required). Defaults to `false`.
|
||||
* __CERTFILE__: HTTPS certificate file path.
|
||||
* __KEYFILE__: HTTPS key file path.
|
||||
* __CORS_ALLOWED_ORIGINS__: Comma-separated list of origins permitted to make cross-origin requests to the MeTube API. When unset or empty, all cross-origin requests are denied. Set to `*` to allow all origins. This must be configured for [browser extensions](#-browser-extensions), [bookmarklets](#-bookmarklet), and any other browser-based tools that contact MeTube from a different origin. For browser extensions use `*` (see below); for bookmarklets you can list specific sites, e.g. `https://www.youtube.com,https://www.vimeo.com`.
|
||||
* __CORS_ALLOWED_ORIGINS__: Comma-separated list of origins permitted to make cross-origin requests to the MeTube API; `*` allows all. When unset or empty, all cross-origin requests are denied. Required for browser extensions and bookmarklets — see [Sending links to MeTube](#-sending-links-to-metube).
|
||||
* __ROBOTS_TXT__: A path to a `robots.txt` file mounted in the container.
|
||||
|
||||
### 🏠 Basic Setup
|
||||
|
||||
* __PUID__: User under which MeTube will run. Defaults to `1000` (legacy `UID` also supported).
|
||||
* __PGID__: Group under which MeTube will run. Defaults to `1000` (legacy `GID` also supported).
|
||||
* __UMASK__: Umask value used by MeTube. Defaults to `022`.
|
||||
* __DEFAULT_THEME__: Default theme to use for the UI, can be set to `light`, `dark`, or `auto`. Defaults to `auto`.
|
||||
* __LOGLEVEL__: Log level, can be set to `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, or `NONE`. Defaults to `INFO`.
|
||||
* __ENABLE_ACCESSLOG__: Whether to enable access log. Defaults to `false`.
|
||||
|
||||
## 🎛️ Configuring yt-dlp options
|
||||
|
||||
MeTube lets you customize how [yt-dlp](https://github.com/yt-dlp/yt-dlp) behaves at three levels, from broadest to most specific:
|
||||
@@ -229,47 +232,27 @@ In case you need to use your browser's cookies with MeTube, for example to downl
|
||||
* After upload, the cookie indicator should show as active.
|
||||
* Use **Delete Cookies** in the same section to remove uploaded cookies.
|
||||
|
||||
## 🔌 Browser extensions
|
||||
## 🔗 Sending links to MeTube
|
||||
|
||||
Browser extensions allow right-clicking videos and sending them directly to MeTube. If you're on an HTTPS page, your MeTube instance must be behind an HTTPS reverse proxy (see below) for extensions to work.
|
||||
Several integrations let you send URLs to MeTube from wherever you are, instead of pasting them into the UI. The browser-based ones make cross-origin requests, so they require `CORS_ALLOWED_ORIGINS` to be set; and if you're on an HTTPS page, your MeTube instance must be served over HTTPS too (with `HTTPS=true` or behind an HTTPS reverse proxy — see below).
|
||||
|
||||
Since browser extensions make requests from their own origin (`chrome-extension://...` or `moz-extension://...`), you must set `CORS_ALLOWED_ORIGINS=*` for them to work.
|
||||
__Browser extensions__ allow right-clicking videos and sending them directly to MeTube. Since extensions request from their own origin, set `CORS_ALLOWED_ORIGINS=*`.
|
||||
* __Chrome:__ contributed by [Rpsl](https://github.com/rpsl) — install from the [Chrome Webstore](https://chrome.google.com/webstore/detail/metube-downloader/fbmkmdnlhacefjljljlbhkodfmfkijdh) or [from sources](https://github.com/Rpsl/metube-browser-extension).
|
||||
* __Firefox:__ contributed by [nanocortex](https://github.com/nanocortex) — install from [Firefox Addons](https://addons.mozilla.org/en-US/firefox/addon/metube-downloader) or get sources [here](https://github.com/nanocortex/metube-firefox-addon).
|
||||
|
||||
__Chrome:__ contributed by [Rpsl](https://github.com/rpsl). You can install it from [Google Chrome Webstore](https://chrome.google.com/webstore/detail/metube-downloader/fbmkmdnlhacefjljljlbhkodfmfkijdh) or use developer mode and install [from sources](https://github.com/Rpsl/metube-browser-extension).
|
||||
__Bookmarklets__ send the currently open page to MeTube with one click. Add the origins of the sites where you use them to `CORS_ALLOWED_ORIGINS`, e.g. `https://www.youtube.com,https://www.vimeo.com`. The code (Chrome and Firefox variants, contributed by [kushfest](https://github.com/kushfest) and [shoonya75](https://github.com/shoonya75)) is in the [Bookmarklets wiki page](https://github.com/alexta69/metube/wiki/Bookmarklets).
|
||||
|
||||
__Firefox:__ contributed by [nanocortex](https://github.com/nanocortex). You can install it from [Firefox Addons](https://addons.mozilla.org/en-US/firefox/addon/metube-downloader) or get sources from [here](https://github.com/nanocortex/metube-firefox-addon).
|
||||
__iOS Shortcut:__ [rithask](https://github.com/rithask) created an [iOS shortcut](https://www.icloud.com/shortcuts/66627a9f334c467baabdb2769763a1a6) for sending URLs to MeTube from Safari's share menu; it prompts for your instance address on first use.
|
||||
|
||||
## 📱 iOS Shortcut
|
||||
__Raycast:__ [dotvhs](https://github.com/dotvhs) has created an [extension for Raycast](https://www.raycast.com/dot/metube) for adding videos to MeTube directly from Raycast.
|
||||
|
||||
[rithask](https://github.com/rithask) created an iOS shortcut to send URLs to MeTube from Safari. Enter the MeTube instance address when prompted which will be saved for later use. You can run the shortcut from Safari’s share menu. The shortcut can be downloaded from [this iCloud link](https://www.icloud.com/shortcuts/66627a9f334c467baabdb2769763a1a6).
|
||||
## 🎵 Pairing with a music tagger
|
||||
|
||||
## 🔖 Bookmarklet
|
||||
MeTube deliberately stops once the file is written — tagging and library organization belong to dedicated tools. Point one at your audio download folder (`AUDIO_DOWNLOAD_DIR`):
|
||||
|
||||
[kushfest](https://github.com/kushfest) has created a Chrome bookmarklet for sending the currently open webpage to MeTube. Please note that if you're on an HTTPS page, your MeTube instance must be configured with `HTTPS` as `true` in the environment, or be behind an HTTPS reverse proxy (see below) for the bookmarklet to work.
|
||||
|
||||
Since bookmarklets run in the context of the current page (e.g. youtube.com), the requests they make to MeTube are cross-origin. You must add the origins of sites where you use the bookmarklet to the __CORS_ALLOWED_ORIGINS__ environment variable, otherwise the browser will block the requests. For example, to use the bookmarklet on YouTube and Vimeo: `CORS_ALLOWED_ORIGINS=https://www.youtube.com,https://www.vimeo.com`.
|
||||
|
||||
GitHub doesn't allow embedding JavaScript as a link, so the bookmarklet has to be created manually by copying the following code to a new bookmark you create on your bookmarks bar. Change the hostname in the URL below to point to your MeTube instance.
|
||||
|
||||
```javascript
|
||||
javascript:!function(){xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.withCredentials=true;xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function(){if(xhr.status==200){alert("Sent to metube!")}else{alert("Send to metube failed. Check the javascript console for clues.")}}}();
|
||||
```
|
||||
|
||||
[shoonya75](https://github.com/shoonya75) has contributed a Firefox version:
|
||||
|
||||
```javascript
|
||||
javascript:(function(){xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function(){if(xhr.status==200){alert("Sent to metube!")}else{alert("Send to metube failed. Check the javascript console for clues.")}}})();
|
||||
```
|
||||
|
||||
The above bookmarklets use `alert()` for notifications. This variant shows a toast instead (Chrome — for Firefox, replace the `!function(){...}()` wrapper with `(function(){...})()`):
|
||||
|
||||
```javascript
|
||||
javascript:!function(){function notify(msg) {var sc = document.scrollingElement.scrollTop; var text = document.createElement('span');text.innerHTML=msg;var ts = text.style;ts.all = 'revert';ts.color = '#000';ts.fontFamily = 'Verdana, sans-serif';ts.fontSize = '15px';ts.backgroundColor = 'white';ts.padding = '15px';ts.border = '1px solid gainsboro';ts.boxShadow = '3px 3px 10px';ts.zIndex = '100';document.body.appendChild(text);ts.position = 'absolute'; ts.top = 50 + sc + 'px'; ts.left = (window.innerWidth / 2)-(text.offsetWidth / 2) + 'px'; setTimeout(function () { text.style.visibility = "hidden"; }, 1500);}xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function() { if(xhr.status==200){notify("Sent to metube!")}else {notify("Send to metube failed. Check the javascript console for clues.")}}}();
|
||||
```
|
||||
|
||||
## ⚡ Raycast extension
|
||||
|
||||
[dotvhs](https://github.com/dotvhs) has created an [extension for Raycast](https://www.raycast.com/dot/metube) for adding videos to MeTube directly from Raycast.
|
||||
* [beets](https://beets.io) — `beet import` matches tracks against MusicBrainz, fixes tags, and files them into an Artist/Album library; headless and scriptable.
|
||||
* [MusicBrainz Picard](https://picard.musicbrainz.org) — GUI tagger with acoustic fingerprinting.
|
||||
* [Lidarr](https://lidarr.audio) — full music library manager; add the folder as an import path.
|
||||
|
||||
## 🔒 HTTPS support, and running behind a reverse proxy
|
||||
|
||||
@@ -293,11 +276,7 @@ services:
|
||||
- KEYFILE=/ssl/key.pem
|
||||
```
|
||||
|
||||
MeTube can also run behind a reverse proxy for HTTPS termination or authentication. When serving under a subdirectory, set `URL_PREFIX` accordingly.
|
||||
|
||||
The [linuxserver/swag](https://docs.linuxserver.io/general/swag) image includes ready-made snippets for MeTube in [subfolder](https://github.com/linuxserver/reverse-proxy-confs/blob/master/metube.subfolder.conf.sample) and [subdomain](https://github.com/linuxserver/reverse-proxy-confs/blob/master/metube.subdomain.conf.sample) modes, plus Authelia for authentication.
|
||||
|
||||
### 🌐 NGINX
|
||||
MeTube can also run behind a reverse proxy for HTTPS termination or authentication. When serving under a subdirectory, set `URL_PREFIX` accordingly. MeTube uses WebSocket for real-time updates, so the proxy must pass the `Upgrade`/`Connection` headers, as in this NGINX example:
|
||||
|
||||
```nginx
|
||||
location /metube/ {
|
||||
@@ -309,41 +288,7 @@ location /metube/ {
|
||||
}
|
||||
```
|
||||
|
||||
Note: the extra `proxy_set_header` directives are there to make WebSocket work.
|
||||
|
||||
### 🌐 Apache
|
||||
|
||||
Contributed by [PIE-yt](https://github.com/PIE-yt). Source [here](https://gist.github.com/PIE-yt/29e7116588379032427f5bd446b2cac4).
|
||||
|
||||
```apache
|
||||
# For putting in your Apache sites site.conf
|
||||
# Serves MeTube under a /metube/ subdir (http://yourdomain.com/metube/)
|
||||
<Location /metube/>
|
||||
ProxyPass http://localhost:8081/ retry=0 timeout=30
|
||||
ProxyPassReverse http://localhost:8081/
|
||||
</Location>
|
||||
|
||||
<Location /metube/socket.io>
|
||||
RewriteEngine On
|
||||
RewriteCond %{QUERY_STRING} transport=websocket [NC]
|
||||
RewriteRule /(.*) ws://localhost:8081/socket.io/$1 [P,L]
|
||||
ProxyPass http://localhost:8081/socket.io retry=0 timeout=30
|
||||
ProxyPassReverse http://localhost:8081/socket.io
|
||||
</Location>
|
||||
```
|
||||
|
||||
### 🌐 Caddy
|
||||
|
||||
The following example Caddyfile gets a reverse proxy going behind [caddy](https://caddyserver.com).
|
||||
|
||||
```caddyfile
|
||||
example.com {
|
||||
route /metube/* {
|
||||
uri strip_prefix metube
|
||||
reverse_proxy metube:8081
|
||||
}
|
||||
}
|
||||
```
|
||||
Apache, Caddy, and [linuxserver/swag](https://docs.linuxserver.io/general/swag) (with Authelia) examples are in the [Reverse proxy configurations wiki page](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations).
|
||||
|
||||
## 🔄 Updating yt-dlp
|
||||
|
||||
@@ -358,9 +303,11 @@ docker exec -ti metube sh
|
||||
cd /downloads
|
||||
```
|
||||
|
||||
Common issues and their fixes are collected in the [Troubleshooting FAQ](https://github.com/alexta69/metube/wiki/Troubleshooting-FAQ) on the wiki.
|
||||
|
||||
## 💡 Submitting feature requests
|
||||
|
||||
MeTube development relies on community contributions. If you need additional features, please submit a PR. Create an issue first to discuss the implementation — some PRs may not be accepted to reduce bloat. Feature requests without an accompanying PR are unlikely to be fulfilled.
|
||||
MeTube development relies on community contributions. If you need additional features, please submit a PR. Create an issue first to discuss the implementation before writing code — MeTube's scope is deliberately narrow: it downloads well and stops once the file is written. Features that improve the download itself are welcome; post-download file management (tag editing, metadata lookups, library organization) is out of scope regardless of implementation quality — see [AGENTS.md](AGENTS.md) for the full policy. Feature requests without an accompanying PR are unlikely to be fulfilled.
|
||||
|
||||
## 🛠️ Building and running locally
|
||||
|
||||
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
# Security Policy
|
||||
|
||||
## Reporting a vulnerability
|
||||
|
||||
Please report vulnerabilities privately via
|
||||
[GitHub private vulnerability reporting](https://github.com/alexta69/metube/security/advisories/new)
|
||||
(Security tab → "Report a vulnerability"). Do **not** open a public issue for
|
||||
security problems.
|
||||
|
||||
You can expect an initial response within a few days. Please include a
|
||||
reproduction and the MeTube release version (visible in the UI footer).
|
||||
|
||||
## Supported versions
|
||||
|
||||
MeTube is continuously released; only the **latest release** is supported.
|
||||
Update to the current Docker image before reporting.
|
||||
|
||||
## Scope notes
|
||||
|
||||
MeTube ships **without authentication by design** — it is intended to run on a
|
||||
trusted network or behind an authenticating reverse proxy (see the
|
||||
[wiki](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations)).
|
||||
Reports that reduce to "the UI is reachable without a login" are expected
|
||||
behavior, not vulnerabilities.
|
||||
+21
-5
@@ -81,6 +81,7 @@ class Config:
|
||||
'YTDL_OPTIONS_PRESETS': '{}',
|
||||
'YTDL_OPTIONS_PRESETS_FILE': '',
|
||||
'ALLOW_YTDL_OPTIONS_OVERRIDES': 'false',
|
||||
'ALLOW_PRIVATE_ADDRESSES': 'false',
|
||||
'CORS_ALLOWED_ORIGINS': '',
|
||||
'ROBOTS_TXT': '',
|
||||
'HOST': '0.0.0.0',
|
||||
@@ -96,7 +97,7 @@ class Config:
|
||||
'YTDL_NIGHTLY_UPDATE_TIME': '',
|
||||
}
|
||||
|
||||
_BOOLEAN = ('DOWNLOAD_DIRS_INDEXABLE', 'CUSTOM_DIRS', 'CREATE_CUSTOM_DIRS', 'DELETE_FILE_ON_TRASHCAN', 'HTTPS', 'ENABLE_ACCESSLOG', 'ALLOW_YTDL_OPTIONS_OVERRIDES')
|
||||
_BOOLEAN = ('DOWNLOAD_DIRS_INDEXABLE', 'CUSTOM_DIRS', 'CREATE_CUSTOM_DIRS', 'DELETE_FILE_ON_TRASHCAN', 'HTTPS', 'ENABLE_ACCESSLOG', 'ALLOW_YTDL_OPTIONS_OVERRIDES', 'ALLOW_PRIVATE_ADDRESSES')
|
||||
|
||||
def __init__(self):
|
||||
for k, v in self._DEFAULTS.items():
|
||||
@@ -717,8 +718,6 @@ def parse_download_options(post: dict) -> dict:
|
||||
|
||||
if custom_name_prefix is None:
|
||||
custom_name_prefix = ''
|
||||
if custom_name_prefix and ('..' in custom_name_prefix or custom_name_prefix.startswith('/') or custom_name_prefix.startswith('\\')):
|
||||
raise web.HTTPBadRequest(reason='custom_name_prefix must not contain ".." or start with a path separator')
|
||||
if auto_start is None:
|
||||
auto_start = True
|
||||
if playlist_item_limit is None:
|
||||
@@ -743,8 +742,6 @@ def parse_download_options(post: dict) -> dict:
|
||||
enabled=config.ALLOW_YTDL_OPTIONS_OVERRIDES,
|
||||
)
|
||||
|
||||
if chapter_template and ('..' in chapter_template or chapter_template.startswith('/') or chapter_template.startswith('\\')):
|
||||
raise web.HTTPBadRequest(reason='chapter_template must not contain ".." or start with a path separator')
|
||||
if not SUBTITLE_LANGUAGE_RE.fullmatch(subtitle_language):
|
||||
raise web.HTTPBadRequest(reason='subtitle_language must match pattern [A-Za-z0-9-] and be at most 35 characters')
|
||||
if subtitle_mode not in VALID_SUBTITLE_MODES:
|
||||
@@ -896,6 +893,17 @@ async def cancel_add(request):
|
||||
return web.Response(text=serializer.encode({'status': 'ok'}), content_type='application/json')
|
||||
|
||||
|
||||
@routes.post(config.URL_PREFIX + 'retry')
|
||||
async def retry(request):
|
||||
# Singular by design, unlike the 'ids' batch endpoints: a retry re-extracts
|
||||
# the URL, so it can fail per item, and the caller removes that item's done
|
||||
# record only once it is confirmed re-queued. A batch form would have to
|
||||
# report per-id results for the caller to know which ones to remove.
|
||||
post = await _read_json_request(request)
|
||||
status = await dqueue.retry(_require_id(post))
|
||||
return web.Response(text=serializer.encode(status), content_type='application/json')
|
||||
|
||||
|
||||
@routes.post(config.URL_PREFIX + 'subscribe')
|
||||
async def subscribe(request):
|
||||
post = await _read_json_request(request)
|
||||
@@ -988,6 +996,13 @@ async def subscriptions_check(request):
|
||||
result = await submgr.check_now([str(i) for i in ids] if ids else None)
|
||||
return web.Response(text=serializer.encode(result))
|
||||
|
||||
def _require_id(post: dict) -> str:
|
||||
id = post.get('id')
|
||||
if not isinstance(id, str) or not id:
|
||||
raise web.HTTPBadRequest(reason="'id' must be a non-empty string")
|
||||
return id
|
||||
|
||||
|
||||
def _require_id_list(post: dict) -> list:
|
||||
ids = post.get('ids')
|
||||
if not isinstance(ids, list) or not ids or not all(isinstance(i, str) for i in ids):
|
||||
@@ -1230,6 +1245,7 @@ async def add_cors(request):
|
||||
|
||||
app.router.add_route('OPTIONS', config.URL_PREFIX + 'add', add_cors)
|
||||
app.router.add_route('OPTIONS', config.URL_PREFIX + 'cancel-add', add_cors)
|
||||
app.router.add_route('OPTIONS', config.URL_PREFIX + 'retry', add_cors)
|
||||
app.router.add_route('OPTIONS', config.URL_PREFIX + 'subscribe', add_cors)
|
||||
app.router.add_route('OPTIONS', config.URL_PREFIX + 'subscriptions', add_cors)
|
||||
app.router.add_route('OPTIONS', config.URL_PREFIX + 'subscriptions/update', add_cors)
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
"""Conservative music metadata enrichment for audio downloads.
|
||||
|
||||
This module only consumes fields already supplied by yt-dlp or retained on
|
||||
MeTube's queued playlist entry. It intentionally performs no external lookup
|
||||
or site-specific album detection.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Optional
|
||||
|
||||
from yt_dlp.postprocessor.common import PostProcessor
|
||||
|
||||
|
||||
def _has_value(value: Any) -> bool:
|
||||
if isinstance(value, str):
|
||||
return bool(value.strip())
|
||||
if isinstance(value, (list, tuple)):
|
||||
return any(_has_value(item) for item in value)
|
||||
return value is not None
|
||||
|
||||
|
||||
def _positive_int(value: Any) -> Optional[int]:
|
||||
if isinstance(value, bool):
|
||||
return None
|
||||
try:
|
||||
number = int(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return number if number > 0 else None
|
||||
|
||||
|
||||
def _track_position(value: Any) -> tuple[Optional[int], Optional[int]]:
|
||||
"""Return a track number and optional total from a scalar or ``n/total``."""
|
||||
if isinstance(value, str) and '/' in value:
|
||||
number, total = value.split('/', 1)
|
||||
return _positive_int(number.strip()), _positive_int(total.strip())
|
||||
return _positive_int(value), None
|
||||
|
||||
|
||||
def _first_positive_int(*values: Any) -> Optional[int]:
|
||||
return next((number for value in values if (number := _positive_int(value))), None)
|
||||
|
||||
|
||||
def _has_album_signal(info: dict[str, Any], source_entry: dict[str, Any]) -> bool:
|
||||
"""Use only extractor-owned fields to identify album-level metadata."""
|
||||
return any(
|
||||
_has_value(entry.get(key))
|
||||
for entry in (info, source_entry)
|
||||
for key in ('album', 'track_number')
|
||||
)
|
||||
|
||||
|
||||
def _is_music_audio(info: dict[str, Any], source_entry: dict[str, Any]) -> bool:
|
||||
return _has_album_signal(info, source_entry) or any(
|
||||
_has_value(entry.get(key))
|
||||
for entry in (info, source_entry)
|
||||
for key in ('track', 'artists')
|
||||
)
|
||||
|
||||
|
||||
def prefer_square_thumbnail(info: dict[str, Any]) -> None:
|
||||
"""Move the largest known square thumbnail to yt-dlp's preferred slot."""
|
||||
thumbnails = info.get('thumbnails')
|
||||
if not isinstance(thumbnails, list) or len(thumbnails) < 2:
|
||||
return
|
||||
|
||||
candidates: list[tuple[int, int]] = []
|
||||
for index, thumbnail in enumerate(thumbnails):
|
||||
if not isinstance(thumbnail, dict):
|
||||
continue
|
||||
width = _positive_int(thumbnail.get('width'))
|
||||
height = _positive_int(thumbnail.get('height'))
|
||||
if width is not None and width == height:
|
||||
candidates.append((width * height, index))
|
||||
if not candidates:
|
||||
return
|
||||
|
||||
_, selected_index = max(candidates)
|
||||
selected = thumbnails.pop(selected_index)
|
||||
thumbnails.append(selected)
|
||||
if selected.get('url'):
|
||||
info['thumbnail'] = selected['url']
|
||||
|
||||
|
||||
class MusicMetadataPreProcessor(PostProcessor):
|
||||
"""Enrich extracted audio metadata using extractor-owned album signals."""
|
||||
|
||||
def __init__(self, downloader=None, *, source_entry=None):
|
||||
super().__init__(downloader)
|
||||
self._source_entry = source_entry if isinstance(source_entry, dict) else {}
|
||||
|
||||
def run(self, info):
|
||||
if _has_album_signal(info, self._source_entry):
|
||||
number, inline_total = _track_position(info.get('track_number'))
|
||||
if number is None:
|
||||
number, source_inline_total = _track_position(
|
||||
self._source_entry.get('track_number')
|
||||
)
|
||||
inline_total = inline_total or source_inline_total
|
||||
if number is None:
|
||||
number = _positive_int(self._source_entry.get('playlist_index'))
|
||||
|
||||
total = inline_total or _first_positive_int(
|
||||
info.get('track_count'),
|
||||
info.get('track_total'),
|
||||
self._source_entry.get('track_count'),
|
||||
self._source_entry.get('track_total'),
|
||||
self._source_entry.get('playlist_count'),
|
||||
self._source_entry.get('n_entries'),
|
||||
)
|
||||
if number is not None:
|
||||
info['track_number'] = f'{number}/{total}' if total is not None else number
|
||||
|
||||
if not _has_value(info.get('album')):
|
||||
album = self._source_entry.get('album') or self._source_entry.get(
|
||||
'playlist_title'
|
||||
)
|
||||
if isinstance(album, str) and album.strip():
|
||||
info['album'] = album.strip()
|
||||
|
||||
if _is_music_audio(info, self._source_entry):
|
||||
prefer_square_thumbnail(info)
|
||||
return [], info
|
||||
+44
-2
@@ -19,6 +19,7 @@ import yt_dlp.networking.impersonate
|
||||
import bg_tasks
|
||||
from dl_formats import merge_ytdl_option_layers
|
||||
from state_store import AtomicJsonStore, read_legacy_shelf
|
||||
from url_guard import validate_url
|
||||
|
||||
log = logging.getLogger("subscriptions")
|
||||
|
||||
@@ -63,6 +64,12 @@ def _build_ydl_params(
|
||||
"paths": {"home": config.DOWNLOAD_DIR, "temp": config.TEMP_DIR},
|
||||
**config.YTDL_OPTIONS,
|
||||
**(extra_opts or {}),
|
||||
# A scan is a poll, not an add: it runs on a timer and queues items
|
||||
# through the download queue, which writes the feed metadata itself.
|
||||
# yt-dlp emits the playlist-level infojson/description/thumbnail
|
||||
# regardless of `download`, so without this a writeinfojson user would
|
||||
# get those files rewritten on every check interval. See issue #1040.
|
||||
"allow_playlist_files": False,
|
||||
}
|
||||
params = _impersonate_opt(params)
|
||||
if playlistend is not None and playlistend > 0:
|
||||
@@ -113,6 +120,9 @@ def extract_flat_playlist(
|
||||
nested_url = _entry_video_url(ent)
|
||||
if not nested_url:
|
||||
continue
|
||||
# nested_url comes from remote playlist content; guard it too.
|
||||
if validate_url(nested_url, allow_private=getattr(config, "ALLOW_PRIVATE_ADDRESSES", False)) is not None:
|
||||
continue
|
||||
nested_info, nested_entries = extract_flat_playlist(
|
||||
config,
|
||||
nested_url,
|
||||
@@ -283,6 +293,24 @@ def validate_title_regex(value: Any) -> str:
|
||||
return s
|
||||
|
||||
|
||||
# The name is a display label the user picks; it is persisted and broadcast to
|
||||
# every connected client, so keep it a bounded single-line string.
|
||||
SUBSCRIPTION_NAME_MAX_LENGTH = 200
|
||||
|
||||
|
||||
def validate_subscription_name(value: Any) -> str:
|
||||
"""Return a stored subscription name, or raise ValueError if unusable."""
|
||||
if not isinstance(value, str):
|
||||
raise ValueError("name must be a string")
|
||||
# Collapse newlines/tabs so a pasted title can't break the table layout.
|
||||
name = " ".join(value.split())
|
||||
if not name:
|
||||
raise ValueError("name must not be empty")
|
||||
if len(name) > SUBSCRIPTION_NAME_MAX_LENGTH:
|
||||
raise ValueError(f"name must be at most {SUBSCRIPTION_NAME_MAX_LENGTH} characters")
|
||||
return name
|
||||
|
||||
|
||||
def _coerce_bool(value: Any) -> bool:
|
||||
"""Accept JSON booleans and common string forms used by API clients."""
|
||||
if isinstance(value, bool):
|
||||
@@ -542,6 +570,13 @@ class SubscriptionManager:
|
||||
url = self._normalize_url(url)
|
||||
if not url:
|
||||
return {"status": "error", "msg": "Missing URL"}
|
||||
# SSRF guard: block non-http(s) schemes and internal/metadata hosts
|
||||
# before yt-dlp fetches the feed. May do a DNS lookup, so run off-loop.
|
||||
url_error = await asyncio.get_running_loop().run_in_executor(
|
||||
None, partial(validate_url, url, allow_private=getattr(self.config, "ALLOW_PRIVATE_ADDRESSES", False)))
|
||||
if url_error is not None:
|
||||
log.warning('Rejected subscription URL "%s": %s', url, url_error)
|
||||
return {"status": "error", "msg": url_error}
|
||||
try:
|
||||
title_regex_stored = validate_title_regex(title_regex)
|
||||
except re.error as exc:
|
||||
@@ -663,6 +698,13 @@ class SubscriptionManager:
|
||||
return {"status": "ok"}
|
||||
|
||||
async def update_subscription(self, sub_id: str, changes: dict) -> dict:
|
||||
validated_name: Optional[str] = None
|
||||
if "name" in changes:
|
||||
try:
|
||||
validated_name = validate_subscription_name(changes["name"])
|
||||
except ValueError as exc:
|
||||
return {"status": "error", "msg": str(exc)}
|
||||
|
||||
validated_tr: Optional[str] = None
|
||||
if "title_regex" in changes:
|
||||
try:
|
||||
@@ -711,8 +753,8 @@ class SubscriptionManager:
|
||||
sub.enabled = validated_enabled
|
||||
if interval_set:
|
||||
sub.check_interval_minutes = validated_interval
|
||||
if "name" in changes and changes["name"]:
|
||||
sub.name = str(changes["name"])
|
||||
if validated_name is not None:
|
||||
sub.name = validated_name
|
||||
if validated_tr is not None:
|
||||
sub.title_regex = validated_tr
|
||||
if skip_so_set:
|
||||
|
||||
+17
-19
@@ -20,6 +20,7 @@ def mock_dqueue(monkeypatch):
|
||||
d = MagicMock()
|
||||
d.initialize = AsyncMock(return_value=None)
|
||||
d.add = AsyncMock(return_value={"status": "ok"})
|
||||
d.retry = AsyncMock(return_value={"status": "ok"})
|
||||
d.cancel = AsyncMock(return_value={"status": "ok"})
|
||||
d.clear = AsyncMock(return_value={"status": "ok"})
|
||||
d.start_pending = AsyncMock(return_value={"status": "ok"})
|
||||
@@ -69,6 +70,22 @@ async def test_add_ok(mock_dqueue):
|
||||
mock_dqueue.add.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retry_passes_failed_download_id(mock_dqueue):
|
||||
req = _json_request({"id": "https://example.com/watch?v=1"})
|
||||
resp = await main.retry(req)
|
||||
assert resp.status == 200
|
||||
mock_dqueue.retry.assert_awaited_once_with("https://example.com/watch?v=1")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("body", [{}, {"id": ""}, {"id": ["a"]}, {"ids": ["a"]}])
|
||||
async def test_retry_rejects_missing_or_non_string_id(mock_dqueue, body):
|
||||
with pytest.raises(web.HTTPBadRequest):
|
||||
await main.retry(_json_request(body))
|
||||
mock_dqueue.retry.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_passes_preset_and_overrides(mock_dqueue, monkeypatch):
|
||||
monkeypatch.setattr(main.config, "YTDL_OPTIONS_PRESETS", {"Preset A": {"writesubtitles": True}})
|
||||
@@ -138,25 +155,6 @@ async def test_add_invalid_subtitle_language(mock_dqueue):
|
||||
await main.add(req)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_custom_name_prefix_path_traversal(mock_dqueue):
|
||||
req = _json_request(_valid_video_add_body(custom_name_prefix="../evil"))
|
||||
with pytest.raises(web.HTTPBadRequest):
|
||||
await main.add(req)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_chapter_template_path_traversal(mock_dqueue):
|
||||
req = _json_request(
|
||||
_valid_video_add_body(
|
||||
split_by_chapters=True,
|
||||
chapter_template="/etc/passwd%(title)s",
|
||||
)
|
||||
)
|
||||
with pytest.raises(web.HTTPBadRequest):
|
||||
await main.add(req)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_invalid_json_body(mock_dqueue):
|
||||
req = MagicMock(spec=web.Request)
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
@@ -89,7 +90,7 @@ def test_get_returns_tuple_of_lists(dq_env):
|
||||
async def test_add_single_video_goes_to_pending_when_auto_start_false(dq_env):
|
||||
notifier = AsyncMock()
|
||||
|
||||
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
|
||||
def fake_extract(self, url, *_args, **_kwargs):
|
||||
return {
|
||||
"_type": "video",
|
||||
"id": "vid1",
|
||||
@@ -115,11 +116,59 @@ async def test_add_single_video_goes_to_pending_when_auto_start_false(dq_env):
|
||||
assert dq.pending.exists("https://example.com/watch?v=1")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_unsupported_url_recorded_as_failed_entry(dq_env):
|
||||
"""An unsupported/unextractable URL must show up as a red-cross entry in the
|
||||
done list, not just a transient toast and a server log line."""
|
||||
import ytdl
|
||||
|
||||
notifier = AsyncMock()
|
||||
url = "https://example.com/not-a-video"
|
||||
|
||||
def boom(self, url, *_args, **_kwargs):
|
||||
raise ytdl.yt_dlp.utils.YoutubeDLError(f'Unsupported URL: {url}')
|
||||
|
||||
dq = DownloadQueue(dq_env, notifier)
|
||||
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", boom):
|
||||
result = await dq.add(
|
||||
url, "video", "auto", "any", "best", "", "", 0, auto_start=True,
|
||||
)
|
||||
assert result["status"] == "error"
|
||||
assert dq.done.exists(url)
|
||||
failed = dq.done.get(url)
|
||||
assert failed.info.status == "error"
|
||||
assert failed.info.error == result["msg"]
|
||||
assert failed.info.url == url
|
||||
# The full URL stays in .url/.error for the detail panel; the display
|
||||
# title is shortened to the hostname so the Completed row stays readable.
|
||||
assert failed.info.title == "example.com"
|
||||
notifier.completed.assert_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_ssrf_rejected_url_recorded_as_failed_entry(dq_env):
|
||||
"""A URL rejected by the SSRF guard (before yt-dlp ever runs) must also
|
||||
surface as a failed entry, not just an error status returned to the caller."""
|
||||
notifier = AsyncMock()
|
||||
url = "file:///etc/passwd"
|
||||
|
||||
dq = DownloadQueue(dq_env, notifier)
|
||||
result = await dq.add(
|
||||
url, "video", "auto", "any", "best", "", "", 0, auto_start=True,
|
||||
)
|
||||
assert result["status"] == "error"
|
||||
assert dq.done.exists(url)
|
||||
failed = dq.done.get(url)
|
||||
assert failed.info.status == "error"
|
||||
assert failed.info.error == result["msg"]
|
||||
notifier.completed.assert_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cancel_removes_from_pending(dq_env):
|
||||
notifier = AsyncMock()
|
||||
|
||||
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
|
||||
def fake_extract(self, url, *_args, **_kwargs):
|
||||
return {
|
||||
"_type": "video",
|
||||
"id": "vid1",
|
||||
@@ -156,7 +205,7 @@ async def test_cancel_before_start_marks_download_canceled(dq_env):
|
||||
cancelling, because its ``download.canceled`` guard was never flipped."""
|
||||
notifier = AsyncMock()
|
||||
|
||||
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
|
||||
def fake_extract(self, url, *_args, **_kwargs):
|
||||
return {
|
||||
"_type": "video",
|
||||
"id": "vid1",
|
||||
@@ -194,7 +243,7 @@ async def test_cancel_before_start_marks_download_canceled(dq_env):
|
||||
async def test_start_pending_moves_to_queue(dq_env):
|
||||
notifier = AsyncMock()
|
||||
|
||||
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
|
||||
def fake_extract(self, url, *_args, **_kwargs):
|
||||
return {
|
||||
"_type": "video",
|
||||
"id": "vid1",
|
||||
@@ -254,6 +303,179 @@ async def test_add_entry_queues_single_video_without_reextracting(dq_env):
|
||||
assert dq.pending.exists("https://example.com/watch?v=1")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retry_restores_playlist_output_context(dq_env):
|
||||
notifier = AsyncMock()
|
||||
dq_env.OUTPUT_TEMPLATE_PLAYLIST = "%(playlist_title)s/%(title)s.%(ext)s"
|
||||
dq = DownloadQueue(dq_env, notifier)
|
||||
url = "https://example.com/watch?v=1"
|
||||
failed_info = DownloadInfo(
|
||||
id="vid1",
|
||||
title="Test Video",
|
||||
url=url,
|
||||
quality="best",
|
||||
download_type="video",
|
||||
codec="auto",
|
||||
format="any",
|
||||
folder="",
|
||||
custom_name_prefix="",
|
||||
error="temporary failure",
|
||||
entry={
|
||||
"playlist_index": "01",
|
||||
"playlist_title": "My Playlist",
|
||||
"playlist_count": 10,
|
||||
},
|
||||
playlist_item_limit=0,
|
||||
split_by_chapters=False,
|
||||
chapter_template="",
|
||||
)
|
||||
failed_info.status = "error"
|
||||
dq.done.put(Download(None, None, None, None, "best", "any", {}, failed_info))
|
||||
|
||||
def fake_extract(self, extracted_url, *_args, **_kwargs):
|
||||
return {
|
||||
"_type": "video",
|
||||
"id": "vid1",
|
||||
"title": "Test Video",
|
||||
"url": extracted_url,
|
||||
"webpage_url": extracted_url,
|
||||
}
|
||||
|
||||
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", fake_extract), \
|
||||
patch.object(DownloadQueue, "_DownloadQueue__start_download", new=AsyncMock()):
|
||||
result = await dq.retry(url)
|
||||
|
||||
assert result["status"] == "ok"
|
||||
queued = dq.queue.get(url)
|
||||
assert queued.output_template == "My Playlist/%(title)s.%(ext)s"
|
||||
assert queued.info.entry["playlist_index"] == "01"
|
||||
assert queued.info.entry["playlist_title"] == "My Playlist"
|
||||
|
||||
|
||||
def _failed_playlist_item(url, **overrides):
|
||||
"""A done-list entry for a playlist item that failed mid-download."""
|
||||
info = DownloadInfo(
|
||||
id="vid1",
|
||||
title="Test Video",
|
||||
url=url,
|
||||
quality="best",
|
||||
download_type="video",
|
||||
codec="auto",
|
||||
format="any",
|
||||
folder="",
|
||||
custom_name_prefix="",
|
||||
error="temporary failure",
|
||||
entry={
|
||||
"playlist_index": "01",
|
||||
"playlist_title": "My Playlist",
|
||||
"playlist_count": 10,
|
||||
},
|
||||
playlist_item_limit=0,
|
||||
split_by_chapters=False,
|
||||
chapter_template="",
|
||||
**overrides,
|
||||
)
|
||||
info.status = "error"
|
||||
return info
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retry_keeps_playlist_context_through_url_indirection(dq_env):
|
||||
# extract_flat=True makes yt-dlp hand back url/url_transparent results
|
||||
# unprocessed, so __add_entry recurses into add() a second time. The retry
|
||||
# context has to survive that hop or the item lands in the root directory.
|
||||
notifier = AsyncMock()
|
||||
dq_env.OUTPUT_TEMPLATE_PLAYLIST = "%(playlist_title)s/%(title)s.%(ext)s"
|
||||
dq = DownloadQueue(dq_env, notifier)
|
||||
url = "https://example.com/watch?v=1"
|
||||
resolved = "https://example.com/resolved?v=1"
|
||||
dq.done.put(Download(None, None, None, None, "best", "any", {}, _failed_playlist_item(url)))
|
||||
|
||||
def fake_extract(self, extracted_url, *_args, **_kwargs):
|
||||
if extracted_url == url:
|
||||
return {"_type": "url", "url": resolved, "id": "vid1"}
|
||||
return {
|
||||
"_type": "video",
|
||||
"id": "vid1",
|
||||
"title": "Test Video",
|
||||
"url": extracted_url,
|
||||
"webpage_url": extracted_url,
|
||||
}
|
||||
|
||||
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", fake_extract), \
|
||||
patch.object(DownloadQueue, "_DownloadQueue__start_download", new=AsyncMock()):
|
||||
result = await dq.retry(url)
|
||||
|
||||
assert result["status"] == "ok"
|
||||
queued = dq.queue.get(resolved)
|
||||
assert queued.output_template == "My Playlist/%(title)s.%(ext)s"
|
||||
assert queued.info.entry["playlist_title"] == "My Playlist"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retry_reapplies_current_options_gates(dq_env):
|
||||
# The stored options passed parse_download_options when first submitted, but
|
||||
# the configuration can have changed since; retry must not resurrect
|
||||
# overrides or presets the current configuration no longer allows.
|
||||
notifier = AsyncMock()
|
||||
dq_env.ALLOW_YTDL_OPTIONS_OVERRIDES = False
|
||||
dq_env.YTDL_OPTIONS_PRESETS = {"Still There": {"writesubtitles": True}}
|
||||
dq = DownloadQueue(dq_env, notifier)
|
||||
url = "https://example.com/watch?v=1"
|
||||
info = _failed_playlist_item(
|
||||
url,
|
||||
ytdl_options_presets=["Still There", "Removed Preset"],
|
||||
ytdl_options_overrides={"paths": {"home": "/etc"}},
|
||||
)
|
||||
dq.done.put(Download(None, None, None, None, "best", "any", {}, info))
|
||||
|
||||
def fake_extract(self, extracted_url, *_args, **_kwargs):
|
||||
return {
|
||||
"_type": "video",
|
||||
"id": "vid1",
|
||||
"title": "Test Video",
|
||||
"url": extracted_url,
|
||||
"webpage_url": extracted_url,
|
||||
}
|
||||
|
||||
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", fake_extract), \
|
||||
patch.object(DownloadQueue, "_DownloadQueue__start_download", new=AsyncMock()):
|
||||
result = await dq.retry(url)
|
||||
|
||||
assert result["status"] == "ok"
|
||||
queued = dq.queue.get(url)
|
||||
assert queued.info.ytdl_options_overrides == {}
|
||||
assert queued.info.ytdl_options_presets == ["Still There"]
|
||||
assert queued.ytdl_opts.get("paths", {}).get("home") != "/etc"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retry_keeps_overrides_while_still_allowed(dq_env):
|
||||
notifier = AsyncMock()
|
||||
dq_env.ALLOW_YTDL_OPTIONS_OVERRIDES = True
|
||||
dq_env.YTDL_OPTIONS_PRESETS = {}
|
||||
dq = DownloadQueue(dq_env, notifier)
|
||||
url = "https://example.com/watch?v=1"
|
||||
info = _failed_playlist_item(url, ytdl_options_overrides={"writesubtitles": True})
|
||||
dq.done.put(Download(None, None, None, None, "best", "any", {}, info))
|
||||
|
||||
def fake_extract(self, extracted_url, *_args, **_kwargs):
|
||||
return {
|
||||
"_type": "video",
|
||||
"id": "vid1",
|
||||
"title": "Test Video",
|
||||
"url": extracted_url,
|
||||
"webpage_url": extracted_url,
|
||||
}
|
||||
|
||||
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", fake_extract), \
|
||||
patch.object(DownloadQueue, "_DownloadQueue__start_download", new=AsyncMock()):
|
||||
result = await dq.retry(url)
|
||||
|
||||
assert result["status"] == "ok"
|
||||
assert dq.queue.get(url).info.ytdl_options_overrides == {"writesubtitles": True}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_entry_duplicate_while_pending_is_skipped_not_clobbered(dq_env):
|
||||
notifier = AsyncMock()
|
||||
@@ -316,7 +538,7 @@ async def test_channel_download_uses_output_template_when_channel_template_empty
|
||||
|
||||
channel_id = "UCabcd123"
|
||||
|
||||
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
|
||||
def fake_extract(self, url, *_args, **_kwargs):
|
||||
return {
|
||||
"_type": "playlist",
|
||||
"id": channel_id,
|
||||
@@ -365,7 +587,7 @@ async def test_playlist_download_not_treated_as_channel(dq_env):
|
||||
dq_env.OUTPUT_TEMPLATE_CHANNEL = ""
|
||||
dq_env.OUTPUT_TEMPLATE_PLAYLIST = "%(playlist_title)s/%(title)s.%(ext)s"
|
||||
|
||||
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
|
||||
def fake_extract(self, url, *_args, **_kwargs):
|
||||
return {
|
||||
"_type": "playlist",
|
||||
"id": "PLxyz789",
|
||||
@@ -412,7 +634,7 @@ async def test_add_merges_global_preset_and_override_options(dq_env):
|
||||
"Preset B": {"writesubtitles": False, "ratelimit": 1000},
|
||||
}
|
||||
|
||||
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
|
||||
def fake_extract(self, url, *_args, **_kwargs):
|
||||
return {
|
||||
"_type": "video",
|
||||
"id": "vid2",
|
||||
@@ -535,11 +757,191 @@ async def test_extract_info_metube_extract_keys_win_over_preset(dq_env):
|
||||
assert captured_params[0]["noplaylist"] is True
|
||||
|
||||
|
||||
def _feed_extract(feed):
|
||||
"""Patch for __extract_info that returns a playlist/channel feed dict."""
|
||||
|
||||
def fake_extract(self, url, *_args, **_kwargs):
|
||||
return copy.deepcopy(feed)
|
||||
|
||||
return fake_extract
|
||||
|
||||
|
||||
_CHANNEL_FEED = {
|
||||
"_type": "playlist",
|
||||
"id": "UC123",
|
||||
"title": "Vanessa - Videos",
|
||||
"channel": "Vanessa",
|
||||
"channel_id": "UC123",
|
||||
"uploader": "Vanessa",
|
||||
"extractor": "youtube:tab",
|
||||
"extractor_key": "YoutubeTab",
|
||||
"webpage_url": "https://example.com/@vanessa/videos",
|
||||
"entries": [
|
||||
{"id": "v1", "title": "One", "url": "https://example.com/v1",
|
||||
"webpage_url": "https://example.com/v1", "_type": "url"},
|
||||
],
|
||||
}
|
||||
|
||||
_PLAYLIST_FEED = {
|
||||
"_type": "playlist",
|
||||
"id": "PL123",
|
||||
"title": "My Playlist",
|
||||
"extractor": "generic",
|
||||
"extractor_key": "Generic",
|
||||
"webpage_url": "https://example.com/playlist?list=PL123",
|
||||
"entries": [
|
||||
{"id": "v1", "title": "One", "url": "https://example.com/v1",
|
||||
"webpage_url": "https://example.com/v1", "_type": "url"},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _written_files(root):
|
||||
found = []
|
||||
for dirpath, _dirs, files in os.walk(root):
|
||||
for f in files:
|
||||
found.append(os.path.relpath(os.path.join(dirpath, f), root))
|
||||
return sorted(found)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_channel_feed_metadata_lands_beside_its_items(dq_env):
|
||||
"""Issues #660/#1040: the feed-level .info.json follows the same template
|
||||
the items use, so it sits in the channel's own folder rather than in
|
||||
DOWNLOAD_DIR under yt-dlp's pl_* default name."""
|
||||
dq_env.YTDL_OPTIONS = {"writeinfojson": True}
|
||||
dq_env.OUTPUT_TEMPLATE_CHANNEL = "%(channel)s/%(title)s.%(ext)s"
|
||||
|
||||
dq = DownloadQueue(dq_env, AsyncMock())
|
||||
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", _feed_extract(_CHANNEL_FEED)), \
|
||||
patch.object(DownloadQueue, "_DownloadQueue__start_download", new=AsyncMock()):
|
||||
result = await dq.add(
|
||||
"https://example.com/@vanessa/videos", "video", "auto", "any", "best",
|
||||
"", "", 0, auto_start=False,
|
||||
)
|
||||
|
||||
assert result["status"] == "ok"
|
||||
assert _written_files(dq_env.DOWNLOAD_DIR) == [
|
||||
os.path.join("Vanessa", "Vanessa - Videos.info.json")
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_playlist_feed_metadata_uses_the_playlist_template(dq_env):
|
||||
dq_env.YTDL_OPTIONS = {"writeinfojson": True}
|
||||
dq_env.OUTPUT_TEMPLATE_PLAYLIST = "%(playlist_title)s/%(title)s.%(ext)s"
|
||||
|
||||
dq = DownloadQueue(dq_env, AsyncMock())
|
||||
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", _feed_extract(_PLAYLIST_FEED)), \
|
||||
patch.object(DownloadQueue, "_DownloadQueue__start_download", new=AsyncMock()):
|
||||
await dq.add(
|
||||
"https://example.com/playlist?list=PL123", "video", "auto", "any", "best",
|
||||
"", "", 0, auto_start=False,
|
||||
)
|
||||
|
||||
assert _written_files(dq_env.DOWNLOAD_DIR) == [
|
||||
os.path.join("My Playlist", "My Playlist.info.json")
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_feed_metadata_honours_custom_folder(dq_env):
|
||||
dq_env.YTDL_OPTIONS = {"writeinfojson": True}
|
||||
dq_env.OUTPUT_TEMPLATE_PLAYLIST = "%(playlist_title)s/%(title)s.%(ext)s"
|
||||
|
||||
dq = DownloadQueue(dq_env, AsyncMock())
|
||||
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", _feed_extract(_PLAYLIST_FEED)), \
|
||||
patch.object(DownloadQueue, "_DownloadQueue__start_download", new=AsyncMock()):
|
||||
await dq.add(
|
||||
"https://example.com/playlist?list=PL123", "video", "auto", "any", "best",
|
||||
"Music", "", 0, auto_start=False,
|
||||
)
|
||||
|
||||
assert _written_files(dq_env.DOWNLOAD_DIR) == [
|
||||
os.path.join("Music", "My Playlist", "My Playlist.info.json")
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_feed_metadata_without_writeinfojson(dq_env):
|
||||
"""Nothing new appears for users who never asked for these files."""
|
||||
dq_env.YTDL_OPTIONS = {}
|
||||
|
||||
dq = DownloadQueue(dq_env, AsyncMock())
|
||||
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", _feed_extract(_PLAYLIST_FEED)), \
|
||||
patch.object(DownloadQueue, "_DownloadQueue__start_download", new=AsyncMock()):
|
||||
await dq.add(
|
||||
"https://example.com/playlist?list=PL123", "video", "auto", "any", "best",
|
||||
"", "", 0, auto_start=False,
|
||||
)
|
||||
|
||||
assert _written_files(dq_env.DOWNLOAD_DIR) == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_feed_metadata_can_be_turned_off_by_the_user(dq_env):
|
||||
dq_env.YTDL_OPTIONS = {"writeinfojson": True, "allow_playlist_files": False}
|
||||
|
||||
dq = DownloadQueue(dq_env, AsyncMock())
|
||||
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", _feed_extract(_PLAYLIST_FEED)), \
|
||||
patch.object(DownloadQueue, "_DownloadQueue__start_download", new=AsyncMock()):
|
||||
await dq.add(
|
||||
"https://example.com/playlist?list=PL123", "video", "auto", "any", "best",
|
||||
"", "", 0, auto_start=False,
|
||||
)
|
||||
|
||||
assert _written_files(dq_env.DOWNLOAD_DIR) == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_feed_metadata_failure_does_not_fail_the_add(dq_env):
|
||||
dq_env.YTDL_OPTIONS = {"writeinfojson": True}
|
||||
|
||||
dq = DownloadQueue(dq_env, AsyncMock())
|
||||
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", _feed_extract(_PLAYLIST_FEED)), \
|
||||
patch.object(DownloadQueue, "_DownloadQueue__start_download", new=AsyncMock()), \
|
||||
patch.object(
|
||||
DownloadQueue, "_DownloadQueue__write_feed_metadata_sync",
|
||||
side_effect=OSError("read-only filesystem"),
|
||||
):
|
||||
result = await dq.add(
|
||||
"https://example.com/playlist?list=PL123", "video", "auto", "any", "best",
|
||||
"", "", 0, auto_start=False,
|
||||
)
|
||||
|
||||
assert result["status"] == "ok"
|
||||
assert dq.pending.exists("https://example.com/v1")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_extraction_pass_never_writes_feed_metadata(dq_env):
|
||||
"""The classification pass must not produce files: it runs before the add is
|
||||
known to succeed, and yt-dlp writes playlist files regardless of `download`."""
|
||||
dq_env.YTDL_OPTIONS = {"writeinfojson": True, "allow_playlist_files": True}
|
||||
captured: list = []
|
||||
|
||||
class FakeYoutubeDL:
|
||||
def __init__(self, params=None):
|
||||
captured.append(params)
|
||||
|
||||
def extract_info(self, url, download=False):
|
||||
return {"_type": "video", "id": "v", "title": "V", "url": url, "webpage_url": url}
|
||||
|
||||
dq = DownloadQueue(dq_env, AsyncMock())
|
||||
with patch("ytdl.yt_dlp.YoutubeDL", FakeYoutubeDL):
|
||||
await dq.add(
|
||||
"https://example.com/watch?v=1", "video", "auto", "any", "best",
|
||||
"", "", 0, auto_start=False,
|
||||
)
|
||||
|
||||
assert captured[0]["allow_playlist_files"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_add_sets_clip_bounds_on_download_info(dq_env):
|
||||
notifier = AsyncMock()
|
||||
|
||||
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
|
||||
def fake_extract(self, url, *_args, **_kwargs):
|
||||
return {
|
||||
"_type": "video",
|
||||
"id": "vid1",
|
||||
@@ -898,6 +1300,40 @@ def _make_download(dq_env, *, download_type="video", status="downloading", filen
|
||||
)
|
||||
|
||||
|
||||
def test_download_close_releases_status_queue(dq_env):
|
||||
download = _make_download(dq_env)
|
||||
status_queue = MagicMock()
|
||||
proc = MagicMock()
|
||||
download.status_queue = status_queue
|
||||
download.proc = proc
|
||||
|
||||
download.close()
|
||||
|
||||
proc.close.assert_called_once()
|
||||
assert download.status_queue is None
|
||||
|
||||
|
||||
def test_download_close_releases_status_queue_without_process(dq_env):
|
||||
download = _make_download(dq_env)
|
||||
download.status_queue = MagicMock()
|
||||
|
||||
download.close()
|
||||
|
||||
assert download.status_queue is None
|
||||
|
||||
|
||||
def test_download_close_releases_status_queue_when_process_close_fails(dq_env):
|
||||
download = _make_download(dq_env)
|
||||
download.status_queue = MagicMock()
|
||||
download.proc = MagicMock()
|
||||
download.proc.close.side_effect = RuntimeError('close failed')
|
||||
|
||||
with pytest.raises(RuntimeError, match='close failed'):
|
||||
download.close()
|
||||
|
||||
assert download.status_queue is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_post_download_cleanup_clears_filename_on_error(dq_env):
|
||||
notifier = AsyncMock()
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
"""Tests for conservative audio metadata enrichment."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from music_metadata import MusicMetadataPreProcessor
|
||||
|
||||
|
||||
def _preprocess(source_entry, info):
|
||||
processor = MusicMetadataPreProcessor(source_entry=source_entry)
|
||||
_, result = processor.run(info)
|
||||
return result
|
||||
|
||||
|
||||
def test_album_uses_existing_order_and_total_when_track_number_is_missing():
|
||||
result = _preprocess(
|
||||
{
|
||||
'playlist_index': '03',
|
||||
'playlist_count': 12,
|
||||
'playlist_title': 'Example Album',
|
||||
},
|
||||
{'title': 'Track', 'album': 'Example Album'},
|
||||
)
|
||||
|
||||
assert result['track_number'] == '3/12'
|
||||
assert result['album'] == 'Example Album'
|
||||
|
||||
|
||||
def test_official_track_number_wins_over_album_order():
|
||||
result = _preprocess(
|
||||
{'playlist_index': 3, 'playlist_count': 12},
|
||||
{'track_number': 7, 'album': 'Official Album'},
|
||||
)
|
||||
|
||||
assert result['track_number'] == '7/12'
|
||||
assert result['album'] == 'Official Album'
|
||||
|
||||
|
||||
def test_inline_official_track_total_is_preserved():
|
||||
result = _preprocess(
|
||||
{'playlist_count': 12},
|
||||
{'track_number': '4/10', 'album': 'Official Album'},
|
||||
)
|
||||
|
||||
assert result['track_number'] == '4/10'
|
||||
|
||||
|
||||
def test_source_track_number_and_total_are_retained_from_flat_extraction():
|
||||
result = _preprocess(
|
||||
{'track_number': 2, 'track_count': 9, 'playlist_index': 4},
|
||||
{'title': 'Track'},
|
||||
)
|
||||
|
||||
assert result['track_number'] == '2/9'
|
||||
|
||||
|
||||
def test_album_title_falls_back_to_source_playlist_title():
|
||||
result = _preprocess(
|
||||
{'playlist_title': 'Example Album'},
|
||||
{'track_number': 4},
|
||||
)
|
||||
|
||||
assert result['album'] == 'Example Album'
|
||||
assert result['track_number'] == 4
|
||||
|
||||
|
||||
def test_playlist_without_extractor_album_signals_is_not_changed():
|
||||
result = _preprocess(
|
||||
{
|
||||
'playlist_index': 3,
|
||||
'playlist_count': 12,
|
||||
'playlist_title': 'Example Playlist',
|
||||
},
|
||||
{'title': 'Track'},
|
||||
)
|
||||
|
||||
assert 'album' not in result
|
||||
assert 'track_number' not in result
|
||||
|
||||
|
||||
def test_regular_video_artwork_is_not_changed():
|
||||
thumbnails = [
|
||||
{'url': 'square.jpg', 'width': 500, 'height': 500},
|
||||
{'url': 'landscape.jpg', 'width': 1280, 'height': 720},
|
||||
]
|
||||
result = _preprocess({}, {'title': 'Regular Video', 'thumbnails': thumbnails.copy()})
|
||||
|
||||
assert result['thumbnails'] == thumbnails
|
||||
assert 'thumbnail' not in result
|
||||
|
||||
|
||||
def test_music_audio_prefers_largest_existing_square_thumbnail():
|
||||
result = _preprocess(
|
||||
{},
|
||||
{
|
||||
'track': 'Track',
|
||||
'thumbnails': [
|
||||
{'url': 'small-square.jpg', 'width': 200, 'height': 200},
|
||||
{'url': 'large-square.jpg', 'width': 1000, 'height': 1000},
|
||||
{'url': 'landscape.jpg', 'width': 1280, 'height': 720},
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
assert result['thumbnails'][-1]['url'] == 'large-square.jpg'
|
||||
assert result['thumbnail'] == 'large-square.jpg'
|
||||
|
||||
|
||||
def test_landscape_only_music_artwork_keeps_existing_order():
|
||||
thumbnails = [
|
||||
{'url': 'small.jpg', 'width': 640, 'height': 360},
|
||||
{'url': 'large.jpg', 'width': 1280, 'height': 720},
|
||||
]
|
||||
result = _preprocess(
|
||||
{},
|
||||
{'track': 'Track', 'thumbnails': thumbnails.copy()},
|
||||
)
|
||||
|
||||
assert result['thumbnails'] == thumbnails
|
||||
assert 'thumbnail' not in result
|
||||
@@ -146,12 +146,12 @@ class PersistentQueueTests(unittest.TestCase):
|
||||
self.assertNotIn("formats", record["entry"])
|
||||
self.assertNotIn("description", record["entry"])
|
||||
|
||||
def test_completed_queue_does_not_persist_entry_or_transient_progress(self):
|
||||
def test_completed_queue_persists_only_failed_retry_context(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = os.path.join(tmp, "completed")
|
||||
pq = PersistentQueue("completed", path)
|
||||
info = _make_info("http://done.example")
|
||||
info.status = "finished"
|
||||
info.status = "error"
|
||||
info.percent = 88
|
||||
info.speed = 123
|
||||
info.eta = 9
|
||||
@@ -167,12 +167,24 @@ class PersistentQueueTests(unittest.TestCase):
|
||||
payload = json.load(f)
|
||||
|
||||
record = payload["items"][0]["info"]
|
||||
self.assertNotIn("entry", record)
|
||||
self.assertEqual(
|
||||
record["entry"],
|
||||
{
|
||||
"playlist_index": "01",
|
||||
"playlist_title": "Playlist",
|
||||
},
|
||||
)
|
||||
self.assertNotIn("percent", record)
|
||||
self.assertNotIn("speed", record)
|
||||
self.assertNotIn("eta", record)
|
||||
self.assertEqual(record["filename"], "done.mp4")
|
||||
|
||||
info.status = "finished"
|
||||
pq.put(_FakeDownload(info))
|
||||
with open(path + ".json", encoding="utf-8") as f:
|
||||
payload = json.load(f)
|
||||
self.assertNotIn("entry", payload["items"][0]["info"])
|
||||
|
||||
def test_invalid_json_is_quarantined_and_legacy_is_imported(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = os.path.join(tmp, "queue")
|
||||
|
||||
@@ -821,6 +821,76 @@ class SubscriptionPersistenceTests(unittest.IsolatedAsyncioTestCase):
|
||||
self.assertEqual(upd["subscription"]["title_regex"], "foo|bar")
|
||||
self.assertEqual(mgr.list_all()[0].title_regex, "foo|bar")
|
||||
|
||||
async def _add_one_subscription(self, mgr):
|
||||
with patch(
|
||||
"subscriptions.extract_flat_playlist",
|
||||
return_value=(
|
||||
{"_type": "channel", "title": "Videos"},
|
||||
[{"id": "v1", "title": "One", "webpage_url": "https://example.com/v1"}],
|
||||
),
|
||||
):
|
||||
result = await mgr.add_subscription(
|
||||
"https://example.com/playlist?list=UULFabc",
|
||||
check_interval_minutes=60,
|
||||
download_type="video",
|
||||
codec="auto",
|
||||
format="any",
|
||||
quality="best",
|
||||
folder="",
|
||||
custom_name_prefix="",
|
||||
auto_start=True,
|
||||
playlist_item_limit=0,
|
||||
split_by_chapters=False,
|
||||
chapter_template="",
|
||||
subtitle_language="en",
|
||||
subtitle_mode="prefer_manual",
|
||||
)
|
||||
return result["subscription"]["id"]
|
||||
|
||||
async def test_update_subscription_renames(self):
|
||||
"""Issue #1044: UULF-style uploads playlists all come back named 'Videos',
|
||||
so the user needs to be able to relabel them."""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
mgr = SubscriptionManager(_Config(tmp), _Queue(), _Notifier())
|
||||
sub_id = await self._add_one_subscription(mgr)
|
||||
self.assertEqual(mgr.list_all()[0].name, "Videos")
|
||||
|
||||
upd = await mgr.update_subscription(sub_id, {"name": " Jane's uploads \n"})
|
||||
self.assertEqual(upd["status"], "ok")
|
||||
# Surrounding and interior whitespace is collapsed to keep the name
|
||||
# a single-line label.
|
||||
self.assertEqual(upd["subscription"]["name"], "Jane's uploads")
|
||||
self.assertEqual(mgr.list_all()[0].name, "Jane's uploads")
|
||||
|
||||
async def test_update_subscription_rename_survives_reload(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
cfg = _Config(tmp)
|
||||
mgr = SubscriptionManager(cfg, _Queue(), _Notifier())
|
||||
sub_id = await self._add_one_subscription(mgr)
|
||||
await mgr.update_subscription(sub_id, {"name": "Renamed"})
|
||||
|
||||
reloaded = SubscriptionManager(cfg, _Queue(), _Notifier())
|
||||
self.assertEqual(reloaded.get(sub_id).name, "Renamed")
|
||||
|
||||
async def test_update_subscription_rejects_unusable_name(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
mgr = SubscriptionManager(_Config(tmp), _Queue(), _Notifier())
|
||||
sub_id = await self._add_one_subscription(mgr)
|
||||
|
||||
for bad in ("", " ", "\n\t", 42, None, ["a"], "x" * 201):
|
||||
upd = await mgr.update_subscription(sub_id, {"name": bad})
|
||||
self.assertEqual(upd["status"], "error", f"expected {bad!r} to be rejected")
|
||||
self.assertEqual(mgr.list_all()[0].name, "Videos")
|
||||
|
||||
async def test_update_subscription_accepts_name_at_length_limit(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
mgr = SubscriptionManager(_Config(tmp), _Queue(), _Notifier())
|
||||
sub_id = await self._add_one_subscription(mgr)
|
||||
|
||||
upd = await mgr.update_subscription(sub_id, {"name": "x" * 200})
|
||||
self.assertEqual(upd["status"], "ok")
|
||||
self.assertEqual(mgr.list_all()[0].name, "x" * 200)
|
||||
|
||||
async def test_update_subscription_skip_subscriber_only(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
queue = _Queue()
|
||||
@@ -1101,6 +1171,44 @@ class SubscriptionScanExtraOptsTests(unittest.IsolatedAsyncioTestCase):
|
||||
self.assertEqual(captured_params[0].get("cookiefile"), "preset.txt")
|
||||
self.assertEqual(captured_params[0].get("extra"), "override")
|
||||
|
||||
async def test_scan_never_writes_playlist_sidecar_files(self):
|
||||
"""A subscription scan is a metadata probe. yt-dlp writes the
|
||||
playlist-level infojson/description/thumbnail regardless of ``download``,
|
||||
so without this a writeinfojson/writethumbnail user would get stray files
|
||||
in DOWNLOAD_DIR on every check interval. Issue #1040."""
|
||||
captured_params: list = []
|
||||
fake_ydl = _make_scan_capturing_fake_ydl(
|
||||
captured_params,
|
||||
[{"id": "v1", "title": "One", "webpage_url": "https://example.com/v1"}],
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
cfg = _Config(tmp)
|
||||
cfg.YTDL_OPTIONS = {"writeinfojson": True, "writethumbnail": True}
|
||||
mgr = SubscriptionManager(cfg, _Queue(), _Notifier())
|
||||
|
||||
with patch("subscriptions.yt_dlp.YoutubeDL", fake_ydl, create=True):
|
||||
await mgr.add_subscription(
|
||||
"https://example.com/channel",
|
||||
check_interval_minutes=60,
|
||||
download_type="video",
|
||||
codec="auto",
|
||||
format="any",
|
||||
quality="best",
|
||||
folder="",
|
||||
custom_name_prefix="",
|
||||
auto_start=True,
|
||||
playlist_item_limit=0,
|
||||
split_by_chapters=False,
|
||||
chapter_template="",
|
||||
subtitle_language="en",
|
||||
subtitle_mode="prefer_manual",
|
||||
ytdl_options_overrides={"allow_playlist_files": True},
|
||||
)
|
||||
|
||||
self.assertTrue(captured_params)
|
||||
self.assertIs(captured_params[0].get("allow_playlist_files"), False)
|
||||
|
||||
async def test_check_now_scan_applies_stored_subscription_presets(self):
|
||||
entries = [{"id": "v1", "title": "One", "webpage_url": "https://example.com/v1"}]
|
||||
|
||||
|
||||
@@ -0,0 +1,283 @@
|
||||
"""Tests for the SSRF URL guard (``url_guard.validate_url``)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
import url_guard
|
||||
from url_guard import (
|
||||
validate_url,
|
||||
_address_allowed_at_connect,
|
||||
_guarded_getaddrinfo,
|
||||
_proxy_endpoint,
|
||||
install_socket_guard,
|
||||
)
|
||||
|
||||
|
||||
def _addrinfo(*addrs, family=socket.AF_INET):
|
||||
return [(family, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", (addr, 0)) for addr in addrs]
|
||||
|
||||
|
||||
class NonUrlInputTests(unittest.TestCase):
|
||||
"""Bare IDs and yt-dlp search/extractor prefixes must pass untouched."""
|
||||
|
||||
def test_bare_video_id_allowed(self):
|
||||
self.assertIsNone(validate_url("dQw4w9WgXcQ"))
|
||||
|
||||
def test_ytsearch_prefix_allowed(self):
|
||||
self.assertIsNone(validate_url("ytsearch:some song"))
|
||||
|
||||
def test_empty_string_allowed(self):
|
||||
self.assertIsNone(validate_url(""))
|
||||
|
||||
def test_non_string_rejected(self):
|
||||
self.assertIsNotNone(validate_url(None))
|
||||
|
||||
|
||||
class SchemeTests(unittest.TestCase):
|
||||
def test_file_scheme_blocked(self):
|
||||
self.assertIsNotNone(validate_url("file:///etc/passwd"))
|
||||
|
||||
def test_ftp_scheme_blocked(self):
|
||||
self.assertIsNotNone(validate_url("ftp://example.com/x"))
|
||||
|
||||
def test_data_scheme_blocked(self):
|
||||
self.assertIsNotNone(validate_url("data://text/plain;base64,AAAA"))
|
||||
|
||||
|
||||
class HostnameBlocklistTests(unittest.TestCase):
|
||||
def test_localhost_blocked_without_lookup(self):
|
||||
with mock.patch("url_guard.socket.getaddrinfo") as gai:
|
||||
self.assertIsNotNone(validate_url("http://localhost:8080/x"))
|
||||
gai.assert_not_called()
|
||||
|
||||
def test_localhost_subdomain_blocked(self):
|
||||
self.assertIsNotNone(validate_url("http://foo.localhost/x"))
|
||||
|
||||
def test_gcp_metadata_name_blocked(self):
|
||||
self.assertIsNotNone(validate_url("http://metadata.google.internal/x"))
|
||||
|
||||
|
||||
class AddressResolutionTests(unittest.TestCase):
|
||||
def _validate_with_addrs(self, url, *addrs, family=socket.AF_INET):
|
||||
with mock.patch("url_guard.socket.getaddrinfo", return_value=_addrinfo(*addrs, family=family)):
|
||||
return validate_url(url)
|
||||
|
||||
def test_public_https_allowed(self):
|
||||
self.assertIsNone(self._validate_with_addrs("https://youtube.com/watch?v=x", "142.250.1.1"))
|
||||
|
||||
def test_public_http_allowed(self):
|
||||
self.assertIsNone(self._validate_with_addrs("http://example.com/x", "93.184.216.34"))
|
||||
|
||||
def test_link_local_metadata_blocked(self):
|
||||
self.assertIsNotNone(self._validate_with_addrs("http://metadata/x", "169.254.169.254"))
|
||||
|
||||
def test_loopback_ipv4_blocked(self):
|
||||
self.assertIsNotNone(self._validate_with_addrs("http://127.0.0.1/x", "127.0.0.1"))
|
||||
|
||||
def test_private_rfc1918_blocked(self):
|
||||
self.assertIsNotNone(self._validate_with_addrs("http://intranet/x", "10.0.0.5"))
|
||||
|
||||
def test_decimal_ip_form_blocked(self):
|
||||
# 2852039166 == 169.254.169.254; the OS resolver normalizes it.
|
||||
self.assertIsNotNone(self._validate_with_addrs("http://2852039166/x", "169.254.169.254"))
|
||||
|
||||
def test_ipv6_loopback_blocked(self):
|
||||
self.assertIsNotNone(
|
||||
self._validate_with_addrs("http://[::1]/x", "::1", family=socket.AF_INET6)
|
||||
)
|
||||
|
||||
def test_ipv4_mapped_ipv6_metadata_blocked(self):
|
||||
self.assertIsNotNone(
|
||||
self._validate_with_addrs(
|
||||
"http://evil/x", "::ffff:169.254.169.254", family=socket.AF_INET6
|
||||
)
|
||||
)
|
||||
|
||||
def test_mixed_public_and_private_blocked(self):
|
||||
# If any resolved address is internal, reject the whole URL.
|
||||
self.assertIsNotNone(self._validate_with_addrs("http://mixed/x", "142.250.1.1", "127.0.0.1"))
|
||||
|
||||
def test_resolution_failure_is_rejected(self):
|
||||
# Fail closed: an unresolvable host cannot be verified as non-internal.
|
||||
with mock.patch("url_guard.socket.getaddrinfo", side_effect=socket.gaierror):
|
||||
self.assertIsNotNone(validate_url("http://does-not-resolve.example/x"))
|
||||
|
||||
|
||||
class ConnectAddressPolicyTests(unittest.TestCase):
|
||||
"""Connect-time policy: allow global, plus loopback only when the caller has
|
||||
established that this destination is the operator's configured proxy."""
|
||||
|
||||
def test_global_allowed(self):
|
||||
self.assertTrue(_address_allowed_at_connect("142.250.1.1"))
|
||||
|
||||
def test_loopback_blocked_by_default(self):
|
||||
# A blanket loopback allowance is what let manifest-derived media URLs
|
||||
# reach services on the server's own loopback interface.
|
||||
self.assertFalse(_address_allowed_at_connect("127.0.0.1"))
|
||||
self.assertFalse(_address_allowed_at_connect("::1"))
|
||||
|
||||
def test_loopback_allowed_only_when_opted_in(self):
|
||||
self.assertTrue(_address_allowed_at_connect("127.0.0.1", allow_loopback=True))
|
||||
self.assertTrue(_address_allowed_at_connect("::1", allow_loopback=True))
|
||||
|
||||
def test_opt_in_does_not_widen_beyond_loopback(self):
|
||||
self.assertFalse(_address_allowed_at_connect("169.254.169.254", allow_loopback=True))
|
||||
self.assertFalse(_address_allowed_at_connect("10.0.0.5", allow_loopback=True))
|
||||
|
||||
def test_link_local_metadata_blocked(self):
|
||||
self.assertFalse(_address_allowed_at_connect("169.254.169.254"))
|
||||
|
||||
def test_private_blocked(self):
|
||||
self.assertFalse(_address_allowed_at_connect("10.0.0.5"))
|
||||
self.assertFalse(_address_allowed_at_connect("192.168.1.10"))
|
||||
|
||||
def test_ipv4_mapped_metadata_blocked(self):
|
||||
self.assertFalse(_address_allowed_at_connect("::ffff:169.254.169.254"))
|
||||
|
||||
|
||||
class ProxyEndpointParsingTests(unittest.TestCase):
|
||||
def test_explicit_port(self):
|
||||
self.assertEqual(_proxy_endpoint("http://127.0.0.1:9050"), ("127.0.0.1", 9050))
|
||||
|
||||
def test_default_port_per_scheme(self):
|
||||
self.assertEqual(_proxy_endpoint("socks5://127.0.0.1"), ("127.0.0.1", 1080))
|
||||
self.assertEqual(_proxy_endpoint("http://127.0.0.1"), ("127.0.0.1", 80))
|
||||
|
||||
def test_bare_host_port(self):
|
||||
self.assertEqual(_proxy_endpoint("127.0.0.1:8080"), ("127.0.0.1", 8080))
|
||||
|
||||
def test_hostname_lowercased(self):
|
||||
self.assertEqual(_proxy_endpoint("http://LocalHost.:9050"), ("localhost", 9050))
|
||||
|
||||
def test_ipv6_literal(self):
|
||||
self.assertEqual(_proxy_endpoint("http://[::1]:9050"), ("::1", 9050))
|
||||
|
||||
def test_empty_and_invalid(self):
|
||||
self.assertIsNone(_proxy_endpoint(""))
|
||||
self.assertIsNone(_proxy_endpoint(" "))
|
||||
self.assertIsNone(_proxy_endpoint(None))
|
||||
self.assertIsNone(_proxy_endpoint("http://"))
|
||||
|
||||
|
||||
class GuardedGetaddrinfoTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
# Default state: no proxy configured, so no loopback destination allowed.
|
||||
saved = set(url_guard._allowed_loopback_endpoints)
|
||||
url_guard._allowed_loopback_endpoints = set()
|
||||
self.addCleanup(lambda: setattr(url_guard, "_allowed_loopback_endpoints", saved))
|
||||
|
||||
def test_internal_only_raises(self):
|
||||
with mock.patch("url_guard._real_getaddrinfo", return_value=_addrinfo("169.254.169.254")):
|
||||
with self.assertRaises(socket.gaierror):
|
||||
_guarded_getaddrinfo("metadata", 80)
|
||||
|
||||
def test_filters_internal_keeps_global(self):
|
||||
# Split-horizon rebinding: keep the public address, drop the internal one.
|
||||
with mock.patch("url_guard._real_getaddrinfo", return_value=_addrinfo("142.250.1.1", "10.0.0.1")):
|
||||
results = _guarded_getaddrinfo("mixed", 80)
|
||||
self.assertEqual([r[4][0] for r in results], ["142.250.1.1"])
|
||||
|
||||
def test_loopback_blocked_without_matching_proxy(self):
|
||||
# The advisory case: an m3u8 segment URL pointing at a loopback service.
|
||||
with mock.patch("url_guard._real_getaddrinfo", return_value=_addrinfo("127.0.0.1")):
|
||||
with self.assertRaises(socket.gaierror):
|
||||
_guarded_getaddrinfo("127.0.0.1", 9999)
|
||||
|
||||
def test_loopback_allowed_at_configured_proxy_endpoint(self):
|
||||
url_guard._allowed_loopback_endpoints = {("127.0.0.1", 9050)}
|
||||
with mock.patch("url_guard._real_getaddrinfo", return_value=_addrinfo("127.0.0.1")):
|
||||
results = _guarded_getaddrinfo("127.0.0.1", 9050)
|
||||
self.assertEqual([r[4][0] for r in results], ["127.0.0.1"])
|
||||
|
||||
def test_loopback_blocked_at_other_port_on_proxy_host(self):
|
||||
# Same host as the proxy, different port: still off limits.
|
||||
url_guard._allowed_loopback_endpoints = {("127.0.0.1", 9050)}
|
||||
with mock.patch("url_guard._real_getaddrinfo", return_value=_addrinfo("127.0.0.1")):
|
||||
with self.assertRaises(socket.gaierror):
|
||||
_guarded_getaddrinfo("127.0.0.1", 9999)
|
||||
|
||||
def test_proxy_reachable_by_hostname(self):
|
||||
url_guard._allowed_loopback_endpoints = {("localhost", 9050)}
|
||||
with mock.patch("url_guard._real_getaddrinfo", return_value=_addrinfo("127.0.0.1")):
|
||||
results = _guarded_getaddrinfo("localhost", 9050)
|
||||
self.assertEqual([r[4][0] for r in results], ["127.0.0.1"])
|
||||
|
||||
def test_string_port_is_normalised(self):
|
||||
url_guard._allowed_loopback_endpoints = {("127.0.0.1", 9050)}
|
||||
with mock.patch("url_guard._real_getaddrinfo", return_value=_addrinfo("127.0.0.1")):
|
||||
results = _guarded_getaddrinfo("127.0.0.1", "9050")
|
||||
self.assertEqual([r[4][0] for r in results], ["127.0.0.1"])
|
||||
|
||||
|
||||
class AllowPrivateBypassTests(unittest.TestCase):
|
||||
"""ALLOW_PRIVATE_ADDRESSES: trusted proxy/VPN environments opt out of the
|
||||
SSRF address checks (e.g. Fake-IP clients that resolve to 198.18.0.0/15)."""
|
||||
|
||||
def test_internal_address_allowed_when_bypassed(self):
|
||||
# Fake-IP benchmarking range that is_global rejects by default.
|
||||
self.assertIsNone(validate_url("http://www.youtube.com/x", allow_private=True))
|
||||
|
||||
def test_private_host_allowed_when_bypassed(self):
|
||||
# No DNS lookup needed: the bypass returns before resolution.
|
||||
with mock.patch("url_guard.socket.getaddrinfo") as gai:
|
||||
self.assertIsNone(validate_url("http://192.168.1.1/x", allow_private=True))
|
||||
gai.assert_not_called()
|
||||
|
||||
def test_scheme_still_enforced_when_bypassed(self):
|
||||
self.assertIsNotNone(validate_url("file:///etc/passwd", allow_private=True))
|
||||
|
||||
def test_socket_guard_not_installed_when_bypassed(self):
|
||||
original = socket.getaddrinfo
|
||||
try:
|
||||
install_socket_guard(allow_private=True)
|
||||
self.assertIs(socket.getaddrinfo, original)
|
||||
finally:
|
||||
socket.getaddrinfo = original
|
||||
|
||||
|
||||
class InstallSocketGuardTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
original, saved = socket.getaddrinfo, set(url_guard._allowed_loopback_endpoints)
|
||||
self.addCleanup(lambda: setattr(socket, "getaddrinfo", original))
|
||||
self.addCleanup(lambda: setattr(url_guard, "_allowed_loopback_endpoints", saved))
|
||||
# Keep the host's own environment out of the assertions below.
|
||||
patcher = mock.patch("url_guard.urllib.request.getproxies", return_value={})
|
||||
self.getproxies = patcher.start()
|
||||
self.addCleanup(patcher.stop)
|
||||
|
||||
def test_install_replaces_and_is_idempotent(self):
|
||||
install_socket_guard()
|
||||
self.assertIs(socket.getaddrinfo, url_guard._guarded_getaddrinfo)
|
||||
# Re-installing must not wrap the wrapper (real fn captured at import).
|
||||
install_socket_guard()
|
||||
self.assertIs(socket.getaddrinfo, url_guard._guarded_getaddrinfo)
|
||||
|
||||
def test_no_proxy_means_no_loopback_allowance(self):
|
||||
install_socket_guard()
|
||||
self.assertEqual(url_guard._allowed_loopback_endpoints, set())
|
||||
|
||||
def test_explicit_proxy_is_registered(self):
|
||||
install_socket_guard(proxy_urls=("socks5://127.0.0.1:9050",))
|
||||
self.assertEqual(url_guard._allowed_loopback_endpoints, {("127.0.0.1", 9050)})
|
||||
|
||||
def test_unset_proxy_option_is_ignored(self):
|
||||
# ytdl_opts.get('proxy') is None when the operator configured no proxy.
|
||||
install_socket_guard(proxy_urls=(None,))
|
||||
self.assertEqual(url_guard._allowed_loopback_endpoints, set())
|
||||
|
||||
def test_environment_proxies_are_registered(self):
|
||||
self.getproxies.return_value = {"http": "http://127.0.0.1:8080"}
|
||||
install_socket_guard()
|
||||
self.assertEqual(url_guard._allowed_loopback_endpoints, {("127.0.0.1", 8080)})
|
||||
|
||||
def test_endpoints_reset_between_installs(self):
|
||||
install_socket_guard(proxy_urls=("http://127.0.0.1:8080",))
|
||||
install_socket_guard(proxy_urls=(None,))
|
||||
self.assertEqual(url_guard._allowed_loopback_endpoints, set())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+315
-20
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import pickle
|
||||
import signal
|
||||
import sys
|
||||
@@ -15,6 +16,8 @@ from unittest.mock import MagicMock, patch
|
||||
fake_yt_dlp = types.ModuleType("yt_dlp")
|
||||
fake_networking = types.ModuleType("yt_dlp.networking")
|
||||
fake_impersonate = types.ModuleType("yt_dlp.networking.impersonate")
|
||||
fake_postprocessor = types.ModuleType("yt_dlp.postprocessor")
|
||||
fake_postprocessor_common = types.ModuleType("yt_dlp.postprocessor.common")
|
||||
fake_utils = types.ModuleType("yt_dlp.utils")
|
||||
|
||||
|
||||
@@ -24,34 +27,200 @@ class _ImpersonateTarget:
|
||||
return value
|
||||
|
||||
|
||||
class _PostProcessor:
|
||||
def __init__(self, downloader=None):
|
||||
self._downloader = downloader
|
||||
|
||||
|
||||
class _YoutubeDL:
|
||||
"""Minimal stand-in so ``_ConfinedYoutubeDL`` can subclass it under the shim.
|
||||
|
||||
``prepare_filename`` is patched per-test; the real containment logic lives in
|
||||
the ``_ConfinedYoutubeDL`` override, which is what the tests exercise.
|
||||
"""
|
||||
|
||||
def __init__(self, params=None, **kwargs):
|
||||
self.params = params or {}
|
||||
|
||||
def prepare_filename(self, *args, **kwargs):
|
||||
return ""
|
||||
|
||||
def add_post_processor(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
|
||||
fake_utils.DownloadError = type("DownloadError", (Exception,), {})
|
||||
fake_yt_dlp.YoutubeDL = _YoutubeDL
|
||||
fake_impersonate.ImpersonateTarget = _ImpersonateTarget
|
||||
fake_networking.impersonate = fake_impersonate
|
||||
fake_postprocessor_common.PostProcessor = _PostProcessor
|
||||
# The inner ``key`` group mirrors the real ``STR_FORMAT_RE_TMPL`` so that
|
||||
# ``_OUTTMPL_FIELD_RE`` (compiled at import time) has the named group that
|
||||
# ``_resolve_outtmpl_fields`` reads via ``match.group('key')``.
|
||||
fake_utils.STR_FORMAT_RE_TMPL = r"(?P<prefix>)%\((?P<has_key>(?P<key>{}))\)(?P<format>[-0-9.]*{})"
|
||||
fake_utils.STR_FORMAT_TYPES = "diouxXeEfFgGcrsa"
|
||||
fake_yt_dlp.networking = fake_networking
|
||||
fake_yt_dlp.postprocessor = fake_postprocessor
|
||||
fake_yt_dlp.utils = fake_utils
|
||||
sys.modules.setdefault("yt_dlp", fake_yt_dlp)
|
||||
sys.modules.setdefault("yt_dlp.networking", fake_networking)
|
||||
sys.modules.setdefault("yt_dlp.networking.impersonate", fake_impersonate)
|
||||
sys.modules.setdefault("yt_dlp.postprocessor", fake_postprocessor)
|
||||
sys.modules.setdefault("yt_dlp.postprocessor.common", fake_postprocessor_common)
|
||||
sys.modules.setdefault("yt_dlp.utils", fake_utils)
|
||||
|
||||
import ytdl
|
||||
from ytdl import (
|
||||
Download,
|
||||
DownloadInfo,
|
||||
MusicMetadataPreProcessor,
|
||||
_compact_persisted_entry,
|
||||
_convert_srt_to_txt_file,
|
||||
_output_dir_escapes,
|
||||
_AlbumArtistPostProcessor,
|
||||
_resolve_outtmpl_fields,
|
||||
_sanitize_entry_for_pickle,
|
||||
_sanitize_path_component,
|
||||
_short_title_for_failed_url,
|
||||
)
|
||||
|
||||
# Detect whether the real yt-dlp is loaded (as opposed to the minimal fake
|
||||
# shim above). _resolve_outtmpl_fields needs YoutubeDL at runtime.
|
||||
_has_real_ytdlp = hasattr(sys.modules.get("yt_dlp"), "YoutubeDL")
|
||||
# shim above). _resolve_outtmpl_fields needs YoutubeDL.evaluate_outtmpl at
|
||||
# runtime, which the shim's YoutubeDL stand-in deliberately does not provide.
|
||||
_has_real_ytdlp = hasattr(
|
||||
getattr(sys.modules.get("yt_dlp"), "YoutubeDL", None), "evaluate_outtmpl"
|
||||
)
|
||||
|
||||
|
||||
class AlbumArtistPostProcessorTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.postprocessor = _AlbumArtistPostProcessor()
|
||||
|
||||
def test_fills_album_artist_from_artist(self):
|
||||
info = {'album': 'CrasH Talk', 'artist': 'ScHoolboy Q'}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'ScHoolboy Q')
|
||||
|
||||
def test_uses_main_artist_for_featured_track(self):
|
||||
info = {
|
||||
'album': 'CrasH Talk',
|
||||
'artists': ['ScHoolboy Q · Travis Scott'],
|
||||
}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'ScHoolboy Q')
|
||||
|
||||
def test_uses_topic_channel_artist_for_joint_album(self):
|
||||
info = {
|
||||
'album': 'Watch the Throne',
|
||||
'artists': ['JAY-Z', 'Kanye West'],
|
||||
'channel': 'JAY-Z & Kanye West - Topic',
|
||||
}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'JAY-Z & Kanye West')
|
||||
|
||||
def test_uses_topic_uploader_and_strips_suffix_for_compilation(self):
|
||||
info = {
|
||||
'album': 'Compilation',
|
||||
'artist': 'Track Artist',
|
||||
'channel': 'Regular Channel',
|
||||
'uploader': 'Various Artists - Topic',
|
||||
}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'Various Artists')
|
||||
|
||||
def test_regular_channel_falls_back_to_main_artist(self):
|
||||
info = {
|
||||
'album': 'Album',
|
||||
'artist': 'Track Artist',
|
||||
'channel': 'Label Channel',
|
||||
}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'Track Artist')
|
||||
|
||||
def test_preserves_explicit_various_artists(self):
|
||||
info = {
|
||||
'album': 'Revenge of the Dreamers III',
|
||||
'artist': 'J. Cole',
|
||||
'album_artist': 'Various Artists',
|
||||
}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'Various Artists')
|
||||
|
||||
def test_preserves_existing_album_artists_list(self):
|
||||
info = {
|
||||
'album': 'Album',
|
||||
'artist': 'Track Artist',
|
||||
'album_artists': ['Album Artist'],
|
||||
}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artists'], ['Album Artist'])
|
||||
self.assertNotIn('album_artist', result)
|
||||
|
||||
def test_uses_first_artist_when_artist_list_has_multiple_entries(self):
|
||||
info = {'album': 'Album', 'artists': ['Main Artist', 'Featured Artist']}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'Main Artist')
|
||||
|
||||
def test_does_not_fill_without_album(self):
|
||||
info = {'artist': 'Standalone Artist'}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertNotIn('album_artist', result)
|
||||
self.assertNotIn('album_artists', result)
|
||||
|
||||
def test_does_not_fill_without_artist(self):
|
||||
info = {'album': 'Instrumental Album'}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertNotIn('album_artist', result)
|
||||
self.assertNotIn('album_artists', result)
|
||||
|
||||
|
||||
class AlbumArtistRegistrationTests(unittest.TestCase):
|
||||
def test_audio_download_registers_pre_process_postprocessor(self):
|
||||
download = _make_test_download()
|
||||
download.info.download_type = 'audio'
|
||||
fake_ydl = MagicMock()
|
||||
|
||||
with patch('ytdl._ConfinedYoutubeDL', return_value=fake_ydl):
|
||||
result = download._make_youtube_dl({'quiet': True})
|
||||
|
||||
self.assertIs(result, fake_ydl)
|
||||
album_artist_call = fake_ydl.add_post_processor.call_args_list[0]
|
||||
postprocessor, = album_artist_call.args
|
||||
self.assertIsInstance(postprocessor, _AlbumArtistPostProcessor)
|
||||
self.assertEqual(album_artist_call.kwargs, {'when': 'pre_process'})
|
||||
metadata_pre_call = fake_ydl.add_post_processor.call_args_list[1]
|
||||
metadata_preprocessor, = metadata_pre_call.args
|
||||
self.assertIsInstance(metadata_preprocessor, MusicMetadataPreProcessor)
|
||||
self.assertEqual(metadata_pre_call.kwargs, {'when': 'pre_process'})
|
||||
self.assertEqual(fake_ydl.add_post_processor.call_count, 2)
|
||||
|
||||
def test_video_download_does_not_register_postprocessor(self):
|
||||
download = _make_test_download()
|
||||
fake_ydl = MagicMock()
|
||||
|
||||
with patch('ytdl._ConfinedYoutubeDL', return_value=fake_ydl):
|
||||
download._make_youtube_dl({'quiet': True})
|
||||
|
||||
fake_ydl.add_post_processor.assert_not_called()
|
||||
|
||||
|
||||
class SanitizePathComponentTests(unittest.TestCase):
|
||||
@@ -162,18 +331,50 @@ class ResolveOuttmplFieldsTests(unittest.TestCase):
|
||||
self.assertFalse(literal_prefix.startswith('\\'))
|
||||
|
||||
|
||||
class OutputDirEscapesTests(unittest.TestCase):
|
||||
class ConfinedYoutubeDLTests(unittest.TestCase):
|
||||
"""The chokepoint: ``_ConfinedYoutubeDL.prepare_filename`` validates the
|
||||
*resolved* output path (after yt-dlp expands the template) and refuses any
|
||||
write outside the allowed roots. This is the single guard for the download-
|
||||
directory invariant across the main file, split-chapter files, thumbnails,
|
||||
subtitles, etc. — the ``..`` only exists post-expansion, so it is caught here
|
||||
rather than by any ingress string check.
|
||||
"""
|
||||
|
||||
def setUp(self):
|
||||
self.base_dir = tempfile.mkdtemp()
|
||||
self.base = os.path.realpath(tempfile.mkdtemp())
|
||||
|
||||
def test_relative_traversal_escapes(self):
|
||||
self.assertTrue(_output_dir_escapes(self.base_dir, '../../tmp/x/%(title)s.%(ext)s'))
|
||||
def _prepared_path(self, resolved, roots=None):
|
||||
ydl = ytdl._ConfinedYoutubeDL.__new__(ytdl._ConfinedYoutubeDL)
|
||||
ydl._allowed_roots = [self.base] if roots is None else roots
|
||||
with patch.object(
|
||||
ytdl.yt_dlp.YoutubeDL, "prepare_filename", return_value=resolved
|
||||
):
|
||||
return ydl.prepare_filename({})
|
||||
|
||||
def test_absolute_path_escapes(self):
|
||||
self.assertTrue(_output_dir_escapes(self.base_dir, '/tmp/x/%(title)s.%(ext)s'))
|
||||
def test_chapter_traversal_via_metadata_is_blocked(self):
|
||||
# e.g. chapter_template '%(section_title)s/%(section_title)s/pwned.%(ext)s'
|
||||
# with a chapter titled '..' expands to '../../pwned.mp4'.
|
||||
escaping = os.path.join(self.base, "..", "..", "pwned.mp4")
|
||||
with self.assertRaises(ytdl.yt_dlp.utils.DownloadError):
|
||||
self._prepared_path(escaping)
|
||||
|
||||
def test_normal_playlist_dir_stays_inside(self):
|
||||
self.assertFalse(_output_dir_escapes(self.base_dir, 'Playlist/%(title)s.%(ext)s'))
|
||||
def test_absolute_output_path_is_blocked(self):
|
||||
with self.assertRaises(ytdl.yt_dlp.utils.DownloadError):
|
||||
self._prepared_path("/etc/cron.d/evil")
|
||||
|
||||
def test_path_inside_download_dir_is_allowed(self):
|
||||
ok = os.path.join(self.base, "Playlist", "video.mp4")
|
||||
self.assertEqual(self._prepared_path(ok), ok)
|
||||
|
||||
def test_sibling_prefix_directory_is_blocked(self):
|
||||
# base '/x/downloads' must not be escapable to '/x/downloads-secret'.
|
||||
sibling = self.base + "-secret"
|
||||
with self.assertRaises(ytdl.yt_dlp.utils.DownloadError):
|
||||
self._prepared_path(os.path.join(sibling, "video.mp4"))
|
||||
|
||||
def test_empty_and_stdout_targets_pass_through(self):
|
||||
self.assertEqual(self._prepared_path(""), "")
|
||||
self.assertEqual(self._prepared_path("-"), "-")
|
||||
|
||||
|
||||
class SanitizeEntryForPickleTests(unittest.TestCase):
|
||||
@@ -267,11 +468,16 @@ class ProgressThrottleTests(unittest.TestCase):
|
||||
|
||||
|
||||
class CancelProcessGroupTests(unittest.TestCase):
|
||||
def test_cancel_kills_group_when_child_is_group_leader(self):
|
||||
# cancel() now sends SIGINT first (so yt-dlp/ffmpeg can finalize the
|
||||
# partial file) and schedules a SIGKILL escalation via the event loop
|
||||
# after ytdl._CANCEL_GRACE_SECONDS, instead of SIGKILLing immediately.
|
||||
|
||||
def test_cancel_sends_sigint_to_group_and_schedules_sigkill_escalation(self):
|
||||
# Child successfully ran os.setpgrp(): its pgid equals its own pid.
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321)
|
||||
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
|
||||
dl.loop = MagicMock()
|
||||
|
||||
with patch.object(Download, "running", return_value=True), \
|
||||
patch("ytdl.os.getpgid", return_value=4321) as mock_getpgid, \
|
||||
@@ -279,38 +485,109 @@ class CancelProcessGroupTests(unittest.TestCase):
|
||||
dl.cancel()
|
||||
|
||||
mock_getpgid.assert_called_once_with(4321)
|
||||
mock_killpg.assert_called_once_with(4321, signal.SIGKILL)
|
||||
mock_killpg.assert_called_once_with(4321, signal.SIGINT)
|
||||
dl.loop.call_later.assert_called_once_with(ytdl._CANCEL_GRACE_SECONDS, dl._kill_if_alive)
|
||||
self.assertTrue(dl.canceled)
|
||||
|
||||
def test_cancel_does_not_killpg_parent_group_kills_child_only(self):
|
||||
def test_cancel_does_not_killpg_parent_group_signals_child_only(self):
|
||||
# Child has NOT become its own group leader yet (pgid != pid, e.g. it is
|
||||
# still in the server's process group). killpg must NOT be called — that
|
||||
# would SIGKILL the whole server — and we fall back to proc.kill().
|
||||
# would signal the whole server — and we fall back to os.kill(pid, SIGINT).
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321, kill=MagicMock())
|
||||
dl.proc = types.SimpleNamespace(pid=4321)
|
||||
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
|
||||
dl.loop = MagicMock()
|
||||
|
||||
with patch.object(Download, "running", return_value=True), \
|
||||
patch("ytdl.os.getpgid", return_value=999), \
|
||||
patch("ytdl.os.killpg") as mock_killpg:
|
||||
patch("ytdl.os.killpg") as mock_killpg, \
|
||||
patch("ytdl.os.kill") as mock_kill:
|
||||
dl.cancel()
|
||||
|
||||
mock_killpg.assert_not_called()
|
||||
dl.proc.kill.assert_called_once()
|
||||
mock_kill.assert_called_once_with(4321, signal.SIGINT)
|
||||
dl.loop.call_later.assert_called_once_with(ytdl._CANCEL_GRACE_SECONDS, dl._kill_if_alive)
|
||||
self.assertTrue(dl.canceled)
|
||||
|
||||
def test_cancel_falls_back_to_proc_kill_when_getpgid_unavailable(self):
|
||||
def test_cancel_falls_back_to_pid_signal_when_getpgid_unavailable(self):
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321)
|
||||
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
|
||||
dl.loop = MagicMock()
|
||||
|
||||
with patch.object(Download, "running", return_value=True), \
|
||||
patch("ytdl.os.getpgid", side_effect=OSError("no such process")), \
|
||||
patch("ytdl.os.kill") as mock_kill:
|
||||
dl.cancel()
|
||||
|
||||
mock_kill.assert_called_once_with(4321, signal.SIGINT)
|
||||
dl.loop.call_later.assert_called_once_with(ytdl._CANCEL_GRACE_SECONDS, dl._kill_if_alive)
|
||||
self.assertTrue(dl.canceled)
|
||||
|
||||
def test_cancel_kills_immediately_when_signal_delivery_fails(self):
|
||||
# Neither killpg nor os.kill succeed (process already gone): cancel()
|
||||
# must fall through to an immediate SIGKILL attempt instead of
|
||||
# scheduling a pointless escalation.
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321, kill=MagicMock())
|
||||
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
|
||||
dl.loop = MagicMock()
|
||||
|
||||
with patch.object(Download, "running", return_value=True), \
|
||||
patch("ytdl.os.getpgid", side_effect=OSError("no such process")):
|
||||
patch("ytdl.os.getpgid", side_effect=OSError("no such process")), \
|
||||
patch("ytdl.os.kill", side_effect=ProcessLookupError()):
|
||||
dl.cancel()
|
||||
|
||||
dl.loop.call_later.assert_not_called()
|
||||
dl.proc.kill.assert_called_once()
|
||||
self.assertTrue(dl.canceled)
|
||||
|
||||
def test_cancel_schedules_escalation_even_without_running_loop(self):
|
||||
# dl.loop is None (e.g. cancel() called before start()'s run_in_executor
|
||||
# set it up): _kill_if_alive() must run synchronously instead of being
|
||||
# scheduled, since there's no loop to schedule it on.
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321)
|
||||
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
|
||||
self.assertIsNone(dl.loop)
|
||||
|
||||
with patch.object(Download, "running", side_effect=[True, True]), \
|
||||
patch("ytdl.os.getpgid", return_value=4321), \
|
||||
patch("ytdl.os.killpg") as mock_killpg:
|
||||
dl.cancel()
|
||||
|
||||
# First SIGINT, then _kill_if_alive() ran inline and sent SIGKILL.
|
||||
mock_killpg.assert_has_calls([
|
||||
unittest.mock.call(4321, signal.SIGINT),
|
||||
unittest.mock.call(4321, signal.SIGKILL),
|
||||
])
|
||||
self.assertTrue(dl.canceled)
|
||||
|
||||
|
||||
class KillIfAliveTests(unittest.TestCase):
|
||||
def test_kill_if_alive_sigkills_running_process(self):
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321)
|
||||
|
||||
with patch.object(Download, "running", return_value=True), \
|
||||
patch("ytdl.os.getpgid", return_value=4321), \
|
||||
patch("ytdl.os.killpg") as mock_killpg:
|
||||
dl._kill_if_alive()
|
||||
|
||||
mock_killpg.assert_called_once_with(4321, signal.SIGKILL)
|
||||
|
||||
def test_kill_if_alive_noop_when_process_already_exited(self):
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321)
|
||||
|
||||
with patch.object(Download, "running", return_value=False), \
|
||||
patch("ytdl.os.killpg") as mock_killpg, \
|
||||
patch("ytdl.os.kill") as mock_kill:
|
||||
dl._kill_if_alive()
|
||||
|
||||
mock_killpg.assert_not_called()
|
||||
mock_kill.assert_not_called()
|
||||
|
||||
|
||||
class ConvertSrtToTxtTests(unittest.TestCase):
|
||||
def test_basic_conversion(self):
|
||||
@@ -532,5 +809,23 @@ class CompactPersistedEntryTests(unittest.TestCase):
|
||||
self.assertIsNone(_compact_persisted_entry({"id": "x", "title": "y"}))
|
||||
|
||||
|
||||
class ShortTitleForFailedUrlTests(unittest.TestCase):
|
||||
def test_uses_hostname_for_a_normal_url(self):
|
||||
self.assertEqual(
|
||||
_short_title_for_failed_url("https://example.com/watch?v=1"),
|
||||
"example.com",
|
||||
)
|
||||
|
||||
def test_falls_back_to_raw_value_when_there_is_no_hostname(self):
|
||||
# file:// URIs and bare search terms/video IDs have no netloc to extract.
|
||||
self.assertEqual(_short_title_for_failed_url("file:///etc/passwd"), "file:///etc/passwd")
|
||||
self.assertEqual(_short_title_for_failed_url("ytsearch:some query"), "ytsearch:some query")
|
||||
|
||||
def test_falls_back_to_raw_value_on_unparseable_input(self):
|
||||
# A malformed IPv6-looking host raises ValueError in urlsplit().hostname.
|
||||
malformed = "https://[::1/watch"
|
||||
self.assertEqual(_short_title_for_failed_url(malformed), malformed)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -0,0 +1,250 @@
|
||||
"""Lightweight SSRF guard for user-submitted URLs.
|
||||
|
||||
MeTube hands user-submitted URLs to yt-dlp, whose generic extractor will fetch
|
||||
any ``http(s)`` URL. Without a guard, an attacker can make the server fetch
|
||||
internal endpoints (cloud metadata services, loopback, RFC1918 hosts, etc.) and
|
||||
have the response saved to the download directory and served back.
|
||||
|
||||
This module provides two layers:
|
||||
|
||||
* ``validate_url`` — a cheap validator applied at every URL ingress.
|
||||
* ``install_socket_guard`` — a connect-time ``getaddrinfo`` guard installed in
|
||||
the download subprocess, which re-validates every resolved address and so
|
||||
covers redirects, DNS rebinding, and media URLs yt-dlp derives from remote
|
||||
metadata — for any backend that resolves through Python's socket module.
|
||||
|
||||
Known limitations — network isolation (e.g. Docker) remains the backstop for
|
||||
all of these:
|
||||
|
||||
* The socket guard is installed only in the download subprocess. Metadata
|
||||
extraction (``ytdl.DownloadQueue.__extract_info``) runs in the main process,
|
||||
where installing a process-wide guard would reject the server's own bind on
|
||||
``HOST=0.0.0.0``. So extraction — which also follows redirects — is covered
|
||||
only by ``validate_url`` at ingress, not at connect time; a redirect from an
|
||||
allowed host to an internal one during extraction is not blocked (a lower-
|
||||
impact, blind SSRF, since the extraction response is not written to disk).
|
||||
* Native resolvers (curl_cffi/libcurl via ``--impersonate``) resolve outside
|
||||
Python's socket module and bypass the connect-time guard entirely.
|
||||
"""
|
||||
|
||||
import ipaddress
|
||||
import logging
|
||||
import socket
|
||||
import urllib.request
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
log = logging.getLogger('url_guard')
|
||||
|
||||
_ALLOWED_SCHEMES = ('http', 'https')
|
||||
|
||||
# Ports to assume when a configured proxy URL omits one, per proxy scheme.
|
||||
_PROXY_DEFAULT_PORTS = {
|
||||
'http': 80,
|
||||
'https': 443,
|
||||
'socks4': 1080,
|
||||
'socks4a': 1080,
|
||||
'socks5': 1080,
|
||||
'socks5h': 1080,
|
||||
}
|
||||
|
||||
# Hostnames that must be blocked without needing a lookup. ``localhost`` and any
|
||||
# subdomain of it are conventionally loopback, and the GCP metadata name is a
|
||||
# well-known SSRF target that may resolve via a resolver we don't control.
|
||||
_BLOCKED_HOSTNAMES = ('localhost', 'metadata.google.internal')
|
||||
|
||||
|
||||
def _hostname_is_blocked(hostname: str) -> bool:
|
||||
host = hostname.rstrip('.').lower()
|
||||
for blocked in _BLOCKED_HOSTNAMES:
|
||||
if host == blocked or host.endswith('.' + blocked):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _normalise_ip(addr: str):
|
||||
"""Parse *addr*, unwrapping IPv4-mapped IPv6 (e.g. ``::ffff:169.254.169.254``)
|
||||
so the embedded IPv4 address is judged on its own merits. Returns ``None``
|
||||
when *addr* is not a valid IP literal."""
|
||||
try:
|
||||
ip = ipaddress.ip_address(addr)
|
||||
except ValueError:
|
||||
return None
|
||||
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
|
||||
ip = ip.ipv4_mapped
|
||||
return ip
|
||||
|
||||
|
||||
def _address_is_global(addr: str) -> bool:
|
||||
ip = _normalise_ip(addr)
|
||||
return ip is not None and ip.is_global
|
||||
|
||||
|
||||
def _address_allowed_at_connect(addr: str, allow_loopback: bool = False) -> bool:
|
||||
"""True if *addr* may be connected to at download time.
|
||||
|
||||
Permits global addresses only. Loopback is permitted just for the specific
|
||||
host:port of an operator-configured proxy (see ``_loopback_endpoint_allowed``),
|
||||
never as a blanket rule: media URLs that yt-dlp derives from a remote manifest
|
||||
are attacker-controlled and reach this policy without passing ``validate_url``,
|
||||
so a general loopback allowance would let a hostile playlist read any service
|
||||
on the server's loopback interface. Blocks link-local (cloud metadata at
|
||||
169.254.169.254), private (RFC1918), unique-local and every other non-global
|
||||
range.
|
||||
"""
|
||||
ip = _normalise_ip(addr)
|
||||
if ip is None:
|
||||
return False
|
||||
return ip.is_global or (allow_loopback and ip.is_loopback)
|
||||
|
||||
|
||||
def _proxy_endpoint(proxy_url: str):
|
||||
"""Parse a proxy URL into a ``(hostname, port)`` pair, or ``None`` if it has
|
||||
no usable host. Used to scope the loopback allowance to that endpoint alone."""
|
||||
if not isinstance(proxy_url, str) or not proxy_url.strip():
|
||||
return None
|
||||
candidate = proxy_url.strip()
|
||||
if '://' not in candidate:
|
||||
# Bare host:port, as accepted by the *_proxy environment variables.
|
||||
candidate = '//' + candidate
|
||||
try:
|
||||
parts = urlsplit(candidate)
|
||||
hostname, port = parts.hostname, parts.port
|
||||
except ValueError:
|
||||
return None
|
||||
if not hostname:
|
||||
return None
|
||||
if port is None:
|
||||
port = _PROXY_DEFAULT_PORTS.get(parts.scheme.lower())
|
||||
return (hostname.rstrip('.').lower(), port)
|
||||
|
||||
|
||||
def _collect_proxy_endpoints(proxy_urls) -> set:
|
||||
"""Endpoints of every proxy this download may legitimately dial: the explicit
|
||||
yt-dlp ``proxy`` option plus the ``*_proxy`` environment variables yt-dlp falls
|
||||
back to. All are operator-configured, unlike the URLs inside fetched media."""
|
||||
candidates = list(proxy_urls) + list(urllib.request.getproxies().values())
|
||||
return {ep for ep in map(_proxy_endpoint, candidates) if ep is not None}
|
||||
|
||||
|
||||
# Captured at import so re-installing the guard never wraps the wrapper.
|
||||
_real_getaddrinfo = socket.getaddrinfo
|
||||
|
||||
# Populated by install_socket_guard; empty means no loopback destination is allowed.
|
||||
_allowed_loopback_endpoints: set = set()
|
||||
|
||||
|
||||
def _normalise_port(port):
|
||||
if isinstance(port, str):
|
||||
try:
|
||||
return int(port)
|
||||
except ValueError:
|
||||
try:
|
||||
return socket.getservbyname(port)
|
||||
except OSError:
|
||||
return None
|
||||
return port
|
||||
|
||||
|
||||
def _loopback_endpoint_allowed(host, port) -> bool:
|
||||
if not _allowed_loopback_endpoints or host is None:
|
||||
return False
|
||||
return (str(host).rstrip('.').lower(), _normalise_port(port)) in _allowed_loopback_endpoints
|
||||
|
||||
|
||||
def _guarded_getaddrinfo(host, *args, **kwargs):
|
||||
results = _real_getaddrinfo(host, *args, **kwargs)
|
||||
# Mirrors getaddrinfo(host, port, ...): port is the first optional argument.
|
||||
port = args[0] if args else kwargs.get('port')
|
||||
allow_loopback = _loopback_endpoint_allowed(host, port)
|
||||
allowed = [r for r in results if _address_allowed_at_connect(r[4][0], allow_loopback)]
|
||||
if not allowed:
|
||||
raise socket.gaierror(f'Refusing to connect to non-global address for host {host!r}')
|
||||
return allowed
|
||||
|
||||
|
||||
def install_socket_guard(allow_private: bool = False, proxy_urls=()) -> None:
|
||||
"""Enforce the no-internal-hosts policy at actual connection time.
|
||||
|
||||
``validate_url`` only checks the *submitted* URL string; yt-dlp then follows
|
||||
HTTP redirects and resolves media URLs from remote metadata without
|
||||
re-validating them. Installing this in the download subprocess re-checks
|
||||
every resolved address at connect time, covering redirects, DNS rebinding and
|
||||
manifest-derived media URLs for any networking backend that resolves through
|
||||
Python's socket module (urllib, requests). Native resolvers — notably
|
||||
curl_cffi/libcurl used by ``--impersonate`` — bypass this and rely on network
|
||||
isolation as the backstop.
|
||||
|
||||
*proxy_urls* are the operator's configured proxies (yt-dlp's ``proxy`` option;
|
||||
the ``*_proxy`` environment variables are picked up automatically). A proxy on
|
||||
loopback is reachable at its own host:port, and nothing else on loopback is.
|
||||
That costs proxied setups nothing: yt-dlp resolves the proxy itself at exactly
|
||||
that host:port, and a media URL is either handed to the proxy unresolved or
|
||||
resolved on its own merits — never inheriting the proxy's allowance.
|
||||
|
||||
When *allow_private* is set (``ALLOW_PRIVATE_ADDRESSES``), the guard is not
|
||||
installed at all, so proxy/VPN setups that route through private or Fake-IP
|
||||
ranges keep working.
|
||||
"""
|
||||
if allow_private:
|
||||
return
|
||||
_allowed_loopback_endpoints.clear()
|
||||
_allowed_loopback_endpoints.update(_collect_proxy_endpoints(proxy_urls))
|
||||
for host, port in sorted(_allowed_loopback_endpoints, key=lambda ep: (ep[0], ep[1] or 0)):
|
||||
ip = _normalise_ip(host)
|
||||
if ip is not None and ip.is_loopback:
|
||||
log.info(f'Allowing connections to configured loopback proxy {host}:{port}')
|
||||
socket.getaddrinfo = _guarded_getaddrinfo
|
||||
|
||||
|
||||
def validate_url(url: str, allow_private: bool = False) -> str | None:
|
||||
"""Return an error message if the URL is disallowed, else ``None``.
|
||||
|
||||
Inputs without a ``://`` scheme separator (bare video IDs, ``ytsearch:``
|
||||
and other yt-dlp search/extractor prefixes) are allowed unchanged so that
|
||||
non-URL entries keep working.
|
||||
|
||||
When *allow_private* is set (``ALLOW_PRIVATE_ADDRESSES``), the internal-host
|
||||
and internal-address checks are skipped so that trusted proxy/VPN setups —
|
||||
e.g. Fake-IP clients that resolve YouTube to ``198.18.0.0/15`` — can be used.
|
||||
Scheme validation (http/https only) still applies.
|
||||
"""
|
||||
if not isinstance(url, str):
|
||||
return 'Invalid URL'
|
||||
|
||||
candidate = url.strip()
|
||||
if '://' not in candidate:
|
||||
# Not an absolute URL: bare video IDs, ytsearch: prefixes, etc.
|
||||
return None
|
||||
|
||||
parts = urlsplit(candidate)
|
||||
scheme = parts.scheme.lower()
|
||||
if scheme not in _ALLOWED_SCHEMES:
|
||||
return f'URL scheme "{parts.scheme}" is not allowed (only http and https)'
|
||||
|
||||
hostname = parts.hostname
|
||||
if not hostname:
|
||||
return 'URL is missing a host'
|
||||
|
||||
if allow_private:
|
||||
# Environment is explicitly trusted: skip the SSRF address checks.
|
||||
return None
|
||||
|
||||
if _hostname_is_blocked(hostname):
|
||||
return f'Refusing to fetch internal host "{hostname}"'
|
||||
|
||||
try:
|
||||
addrinfo = socket.getaddrinfo(hostname, parts.port, proto=socket.IPPROTO_TCP)
|
||||
except socket.gaierror:
|
||||
# Fail closed: a host we cannot resolve is a host we cannot verify as
|
||||
# non-internal, so refuse it rather than letting the download proceed
|
||||
# to a target that may resolve differently at fetch time.
|
||||
return f'Could not resolve host "{hostname}"'
|
||||
except (UnicodeError, ValueError):
|
||||
return f'Invalid host "{hostname}"'
|
||||
|
||||
for family, _type, _proto, _canonname, sockaddr in addrinfo:
|
||||
addr = sockaddr[0]
|
||||
if not _address_is_global(addr):
|
||||
return f'Refusing to fetch internal address "{addr}" for host "{hostname}"'
|
||||
|
||||
return None
|
||||
+392
-44
@@ -19,12 +19,16 @@ import types
|
||||
from typing import Any, Optional
|
||||
|
||||
import yt_dlp.networking.impersonate
|
||||
from yt_dlp.postprocessor.common import PostProcessor
|
||||
from yt_dlp.utils import STR_FORMAT_RE_TMPL, STR_FORMAT_TYPES
|
||||
import bg_tasks
|
||||
from dl_formats import get_format, get_opts, AUDIO_FORMATS, merge_ytdl_option_layers
|
||||
from music_metadata import MusicMetadataPreProcessor
|
||||
from datetime import datetime
|
||||
from state_store import AtomicJsonStore, from_json_compatible, read_legacy_shelf, to_json_compatible
|
||||
from subscriptions import _entry_id
|
||||
from url_guard import validate_url, install_socket_guard
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
log = logging.getLogger('ytdl')
|
||||
|
||||
@@ -45,6 +49,10 @@ _MP_CTX = (
|
||||
else multiprocessing.get_context()
|
||||
)
|
||||
|
||||
# Grace period between SIGINT (lets yt-dlp/ffmpeg finalize the partial file,
|
||||
# e.g. when cancelling a livestream) and SIGKILL escalation.
|
||||
_CANCEL_GRACE_SECONDS = 15
|
||||
|
||||
_LIVE_CHECK_INTERVAL = 60
|
||||
_LIVE_MAX_CHECK_INTERVAL = 3600
|
||||
# Consecutive probe failures (network blips, rate limits, transient extractor
|
||||
@@ -52,6 +60,52 @@ _LIVE_MAX_CHECK_INTERVAL = 3600
|
||||
_LIVE_PROBE_MAX_FAILURES = 5
|
||||
|
||||
|
||||
class _AlbumArtistPostProcessor(PostProcessor):
|
||||
"""Fill missing album-artist metadata from yt-dlp's album-level signals."""
|
||||
|
||||
_TOPIC_SUFFIX = ' - Topic'
|
||||
|
||||
@staticmethod
|
||||
def _has_value(value: Any) -> bool:
|
||||
if isinstance(value, str):
|
||||
return bool(value.strip())
|
||||
if isinstance(value, (list, tuple)):
|
||||
return any(_AlbumArtistPostProcessor._has_value(item) for item in value)
|
||||
return value is not None
|
||||
|
||||
@staticmethod
|
||||
def _main_artist(info) -> Optional[str]:
|
||||
artists = info.get('artists')
|
||||
candidates = artists if isinstance(artists, list) else [info.get('artist')]
|
||||
for candidate in candidates:
|
||||
if not isinstance(candidate, str) or not candidate.strip():
|
||||
continue
|
||||
# YouTube Music uses a spaced middle dot between credited artists.
|
||||
# The first credit is the primary artist for normal albums.
|
||||
return candidate.split(' · ', 1)[0].strip()
|
||||
return None
|
||||
|
||||
@classmethod
|
||||
def _topic_artist(cls, info) -> Optional[str]:
|
||||
for field in ('channel', 'uploader'):
|
||||
value = info.get(field)
|
||||
if not isinstance(value, str) or not value.endswith(cls._TOPIC_SUFFIX):
|
||||
continue
|
||||
if artist := value[:-len(cls._TOPIC_SUFFIX)].strip():
|
||||
return artist
|
||||
return None
|
||||
|
||||
def run(self, info):
|
||||
if not self._has_value(info.get('album')):
|
||||
return [], info
|
||||
if self._has_value(info.get('album_artist')) or self._has_value(info.get('album_artists')):
|
||||
return [], info
|
||||
|
||||
if artist := self._topic_artist(info) or self._main_artist(info):
|
||||
info['album_artist'] = artist
|
||||
return [], info
|
||||
|
||||
|
||||
def _is_within_directory(real_base: str, real_target: str) -> bool:
|
||||
"""True if ``real_target`` is inside (or equal to) ``real_base``.
|
||||
|
||||
@@ -93,16 +147,36 @@ def _sanitize_path_component(value: Any) -> Any:
|
||||
return value.lstrip('.').strip() or '_'
|
||||
|
||||
|
||||
def _output_dir_escapes(base_dir: str, output_template: str) -> bool:
|
||||
"""True when the literal directory prefix of *output_template* resolves outside *base_dir*."""
|
||||
marker = output_template.find('%(')
|
||||
literal = output_template if marker == -1 else output_template[:marker]
|
||||
dir_prefix = os.path.dirname(literal)
|
||||
if not dir_prefix:
|
||||
return False
|
||||
real_base = os.path.realpath(base_dir)
|
||||
real_target = os.path.realpath(os.path.join(base_dir, dir_prefix))
|
||||
return not _is_within_directory(real_base, real_target)
|
||||
class _ConfinedYoutubeDL(yt_dlp.YoutubeDL):
|
||||
"""A ``YoutubeDL`` that refuses to emit any output path outside the allowed roots.
|
||||
|
||||
This is the single authoritative enforcement of MeTube's download-directory
|
||||
containment invariant. yt-dlp expands output templates at download time using
|
||||
metadata that is fully attacker-controlled (``%(title)s``, ``%(uploader)s``,
|
||||
``%(section_title)s`` from chapter titles, …) and, on POSIX hosts, does *not*
|
||||
neutralise a ``..`` path component — so any template segment resolving to
|
||||
``..`` next to a literal separator (or an absolute template) can traverse out
|
||||
of the download directory. Every output path — main file, split-chapter files,
|
||||
thumbnails, subtitles, infojson — is produced by ``prepare_filename``, so
|
||||
validating its result here covers them all, regardless of which template or
|
||||
metadata field carries the traversal. Checking the resolved path (rather than
|
||||
the template string) is what makes this robust: the ``..`` only exists after
|
||||
expansion, so no ingress string check can see it.
|
||||
"""
|
||||
|
||||
def __init__(self, params=None, *, allowed_roots=(), **kwargs):
|
||||
self._allowed_roots = [os.path.realpath(r) for r in allowed_roots if r]
|
||||
super().__init__(params=params, **kwargs)
|
||||
|
||||
def prepare_filename(self, *args, **kwargs):
|
||||
filename = super().prepare_filename(*args, **kwargs)
|
||||
if filename and filename != '-' and self._allowed_roots:
|
||||
resolved = os.path.realpath(filename)
|
||||
if not any(_is_within_directory(root, resolved) for root in self._allowed_roots):
|
||||
raise yt_dlp.utils.DownloadError(
|
||||
f'Refusing to write outside the download directory: {filename}'
|
||||
)
|
||||
return filename
|
||||
|
||||
|
||||
# Regex matching yt-dlp output-template field references, e.g. ``%(title)s``
|
||||
@@ -425,7 +499,18 @@ _PERSISTED_DOWNLOAD_FIELDS = (
|
||||
)
|
||||
|
||||
|
||||
_COMPACT_ENTRY_EXTRA_KEYS = frozenset(("n_entries", "__last_playlist_index"))
|
||||
def _short_title_for_failed_url(url: str) -> str:
|
||||
"""A concise display title for a URL that failed before yt-dlp could extract a
|
||||
real title (unsupported URL, SSRF-rejected, extraction error). The full URL
|
||||
remains available in DownloadInfo.url and the error-detail panel."""
|
||||
try:
|
||||
hostname = urlsplit(url).hostname
|
||||
except ValueError:
|
||||
hostname = None
|
||||
return hostname or url
|
||||
|
||||
|
||||
_COMPACT_ENTRY_EXTRA_KEYS = frozenset(("n_entries", "__last_playlist_index", "track_number"))
|
||||
|
||||
|
||||
def _compact_persisted_entry(entry: Any) -> Optional[dict[str, Any]]:
|
||||
@@ -485,11 +570,12 @@ class Download:
|
||||
cls.manager.shutdown()
|
||||
cls.manager = None
|
||||
|
||||
def __init__(self, download_dir, temp_dir, output_template, output_template_chapter, quality, format, ytdl_opts, info):
|
||||
def __init__(self, download_dir, temp_dir, output_template, output_template_chapter, quality, format, ytdl_opts, info, allow_private=False):
|
||||
self.download_dir = download_dir
|
||||
self.temp_dir = temp_dir
|
||||
self.output_template = output_template
|
||||
self.output_template_chapter = output_template_chapter
|
||||
self.allow_private = allow_private
|
||||
self.info = info
|
||||
self.format = get_format(
|
||||
getattr(info, 'download_type', 'video'),
|
||||
@@ -545,6 +631,22 @@ class Download:
|
||||
|
||||
return put_status
|
||||
|
||||
def _make_youtube_dl(self, params):
|
||||
ydl = _ConfinedYoutubeDL(
|
||||
params=params,
|
||||
allowed_roots=(self.download_dir, self.temp_dir),
|
||||
)
|
||||
if getattr(self.info, 'download_type', '') == 'audio':
|
||||
ydl.add_post_processor(_AlbumArtistPostProcessor(ydl), when='pre_process')
|
||||
ydl.add_post_processor(
|
||||
MusicMetadataPreProcessor(
|
||||
ydl,
|
||||
source_entry=getattr(self.info, 'entry', None),
|
||||
),
|
||||
when='pre_process',
|
||||
)
|
||||
return ydl
|
||||
|
||||
def _download(self):
|
||||
# Run in our own process group so cancel() can SIGKILL the whole
|
||||
# group (yt-dlp + any ffmpeg children it spawned for merge/postproc),
|
||||
@@ -554,6 +656,13 @@ class Download:
|
||||
os.setpgrp()
|
||||
except OSError:
|
||||
pass
|
||||
# Re-validate every outbound connection at fetch time. validate_url only
|
||||
# saw the submitted URL string; this catches redirects, DNS rebinding and
|
||||
# attacker-controlled media URLs pulled from a remote manifest, none of
|
||||
# which it can see. The configured proxy is passed so that a proxy on
|
||||
# loopback stays reachable at its own address without opening up the rest
|
||||
# of loopback. Skipped when ALLOW_PRIVATE_ADDRESSES trusts the environment.
|
||||
install_socket_guard(self.allow_private, proxy_urls=(self.ytdl_opts.get('proxy'),))
|
||||
log.info(f"Starting download for: {self.info.title} ({self.info.url})")
|
||||
try:
|
||||
debug_logging = logging.getLogger().isEnabledFor(logging.DEBUG)
|
||||
@@ -622,7 +731,7 @@ class Download:
|
||||
[(start, end)],
|
||||
)
|
||||
|
||||
ret = yt_dlp.YoutubeDL(params=ytdl_params).download([self.info.url])
|
||||
ret = self._make_youtube_dl(ytdl_params).download([self.info.url])
|
||||
self.status_queue.put({'status': 'finished' if ret == 0 else 'error'})
|
||||
log.info(f"Finished download for: {self.info.title}")
|
||||
except yt_dlp.utils.YoutubeDLError as exc:
|
||||
@@ -650,37 +759,61 @@ class Download:
|
||||
self.status_queue.put(None)
|
||||
await self.status_task
|
||||
|
||||
def _signal_group(self, sig):
|
||||
"""Send *sig* to the download's process group, falling back to the
|
||||
process itself. Returns True if a signal was delivered.
|
||||
|
||||
Only signal the whole group when the child actually became its own
|
||||
group leader via os.setpgrp() in _download() — that sets its pgid
|
||||
equal to its own pid. If it hasn't run setpgrp() yet, or setpgrp()
|
||||
failed, its pgid is still the SERVER's group and killpg would signal
|
||||
the entire MeTube process (PID 1 in Docker). Fall back to signalling
|
||||
just the child process by pid in that case.
|
||||
"""
|
||||
try:
|
||||
pgid = os.getpgid(self.proc.pid)
|
||||
if pgid == self.proc.pid:
|
||||
os.killpg(pgid, sig)
|
||||
return True
|
||||
except (OSError, AttributeError):
|
||||
pass
|
||||
try:
|
||||
if sig == signal.SIGINT:
|
||||
os.kill(self.proc.pid, sig)
|
||||
else:
|
||||
self.proc.kill()
|
||||
return True
|
||||
except Exception as e:
|
||||
log.error(f"Error signalling process for {self.info.title}: {e}")
|
||||
return False
|
||||
|
||||
def _kill_if_alive(self):
|
||||
if self.running():
|
||||
log.info(f"Escalating cancel to SIGKILL for: {self.info.title}")
|
||||
self._signal_group(signal.SIGKILL)
|
||||
|
||||
def cancel(self):
|
||||
log.info(f"Cancelling download: {self.info.title}")
|
||||
if self.running():
|
||||
killed_group = False
|
||||
try:
|
||||
pgid = os.getpgid(self.proc.pid)
|
||||
# Only kill the whole group (yt-dlp + any ffmpeg children) when
|
||||
# the child actually became its own group leader via
|
||||
# os.setpgrp() in _download() — that sets its pgid equal to its
|
||||
# own pid. If it hasn't run setpgrp() yet, or setpgrp() failed,
|
||||
# its pgid is still the SERVER's group and killpg would SIGKILL
|
||||
# the entire MeTube process (PID 1 in Docker). Fall back to
|
||||
# killing just the child process by pid in that case.
|
||||
if pgid == self.proc.pid:
|
||||
os.killpg(pgid, signal.SIGKILL)
|
||||
killed_group = True
|
||||
except (OSError, AttributeError):
|
||||
pass
|
||||
if not killed_group:
|
||||
try:
|
||||
self.proc.kill()
|
||||
except Exception as e:
|
||||
log.error(f"Error killing process for {self.info.title}: {e}")
|
||||
# SIGINT first so yt-dlp/ffmpeg can finalize the partial file
|
||||
# (livestream recordings stay playable); SIGKILL after a grace
|
||||
# period if the process ignores it.
|
||||
interrupted = self._signal_group(signal.SIGINT)
|
||||
if interrupted and self.loop is not None:
|
||||
self.loop.call_later(_CANCEL_GRACE_SECONDS, self._kill_if_alive)
|
||||
else:
|
||||
self._kill_if_alive()
|
||||
self.canceled = True
|
||||
if self.status_queue is not None:
|
||||
self.status_queue.put(None)
|
||||
|
||||
def close(self):
|
||||
log.info(f"Closing download process for: {self.info.title}")
|
||||
if self.started():
|
||||
self.proc.close()
|
||||
try:
|
||||
if self.started():
|
||||
self.proc.close()
|
||||
finally:
|
||||
self.status_queue = None
|
||||
|
||||
def running(self):
|
||||
try:
|
||||
@@ -813,8 +946,12 @@ class PersistentQueue:
|
||||
]
|
||||
return sorted(items, key=lambda item: item[1].timestamp)
|
||||
|
||||
def _should_persist_entry(self) -> bool:
|
||||
return self.identifier != "completed"
|
||||
def _should_persist_entry(self, info: DownloadInfo | dict[str, Any]) -> bool:
|
||||
# Failed downloads need their compact playlist/channel context so a
|
||||
# retry after a server restart still resolves the original outtmpl.
|
||||
# Successful completed entries continue to omit extractor metadata.
|
||||
status = info.get("status") if isinstance(info, dict) else info.status
|
||||
return self.identifier != "completed" or status == "error"
|
||||
|
||||
def _serialize_items(self):
|
||||
return [
|
||||
@@ -822,7 +959,7 @@ class PersistentQueue:
|
||||
"key": key,
|
||||
"info": _download_info_to_record(
|
||||
download.info,
|
||||
include_entry=self._should_persist_entry(),
|
||||
include_entry=self._should_persist_entry(download.info),
|
||||
),
|
||||
}
|
||||
for key, download in self.dict.items()
|
||||
@@ -841,7 +978,7 @@ class PersistentQueue:
|
||||
"key": item["key"],
|
||||
"info": _download_info_to_record(
|
||||
_download_info_from_record(item["info"]),
|
||||
include_entry=self._should_persist_entry(),
|
||||
include_entry=self._should_persist_entry(item["info"]),
|
||||
),
|
||||
}
|
||||
for item in items
|
||||
@@ -862,7 +999,7 @@ class PersistentQueue:
|
||||
"key": key,
|
||||
"info": _download_info_to_record(
|
||||
value,
|
||||
include_entry=self._should_persist_entry(),
|
||||
include_entry=self._should_persist_entry(value),
|
||||
),
|
||||
}
|
||||
for key, value in sorted(legacy_items, key=lambda item: item[1].timestamp)
|
||||
@@ -1163,6 +1300,11 @@ class DownloadQueue:
|
||||
return opts
|
||||
|
||||
def __extract_info(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
|
||||
# NOTE: extraction runs in the main process, so the connect-time socket
|
||||
# guard (installed only in the download subprocess) does not apply here.
|
||||
# The ingress validate_url check guards the submitted URL, but redirects
|
||||
# followed during extraction are not re-validated. See url_guard's module
|
||||
# docstring for why the guard can't be installed process-wide.
|
||||
debug_logging = logging.getLogger().isEnabledFor(logging.DEBUG)
|
||||
user_opts = self._build_ytdl_options(ytdl_options_presets, ytdl_options_overrides)
|
||||
params = {
|
||||
@@ -1174,6 +1316,14 @@ class DownloadQueue:
|
||||
'ignore_no_formats_error': True,
|
||||
'noplaylist': True,
|
||||
'paths': {"home": self.config.DOWNLOAD_DIR, "temp": self.config.TEMP_DIR},
|
||||
# This is a classification pass, not a download. yt-dlp emits the
|
||||
# feed-level infojson/description/thumbnail from
|
||||
# __process_playlist_result without consulting `download`, so
|
||||
# without this a writeinfojson user gets stray files here — in
|
||||
# DOWNLOAD_DIR, under yt-dlp's pl_* names, even for an add that goes
|
||||
# on to fail. __write_feed_metadata writes them properly once the
|
||||
# feed is accepted. See issues #1040 and #660.
|
||||
'allow_playlist_files': False,
|
||||
}
|
||||
imp = user_opts.get('impersonate')
|
||||
if imp is not None:
|
||||
@@ -1222,9 +1372,7 @@ class DownloadQueue:
|
||||
if playlist_item_limit > 0:
|
||||
log.info(f'playlist limit is set. Processing only first {playlist_item_limit} entries')
|
||||
ytdl_options['playlistend'] = playlist_item_limit
|
||||
if _output_dir_escapes(dldirectory, output):
|
||||
return {'status': 'error', 'msg': 'Refusing download: resolved output path escapes the download directory'}
|
||||
download = Download(dldirectory, self.config.TEMP_DIR, output, output_chapter, dl.quality, dl.format, ytdl_options, dl)
|
||||
download = Download(dldirectory, self.config.TEMP_DIR, output, output_chapter, dl.quality, dl.format, ytdl_options, dl, allow_private=self.config.ALLOW_PRIVATE_ADDRESSES)
|
||||
is_upcoming = (
|
||||
getattr(dl, 'live_status', None) == 'is_upcoming'
|
||||
or getattr(dl, 'status', None) == 'scheduled'
|
||||
@@ -1239,6 +1387,81 @@ class DownloadQueue:
|
||||
self.pending.put(download)
|
||||
await self.notifier.added(dl)
|
||||
|
||||
def __write_feed_metadata_sync(self, entry, etype, download_type, folder,
|
||||
ytdl_options_presets, ytdl_options_overrides):
|
||||
"""Write the feed-level .info.json/description/thumbnail for a playlist
|
||||
or channel add, using the same output template its items will use.
|
||||
|
||||
yt-dlp produces these from __process_playlist_result, which ignores
|
||||
``download`` — so they used to fall out of the classification pass with
|
||||
yt-dlp's own pl_* names, in DOWNLOAD_DIR, ignoring the download's folder
|
||||
(issue #1040) and with no way to steer them (issue #660). Doing it here
|
||||
instead means the feed type is already known, so the file lands beside
|
||||
the items rather than in a differently-named sibling directory.
|
||||
|
||||
Re-runs yt-dlp on a copy of the feed with no entries: that reaches the
|
||||
playlist-file writing without re-extracting anything or touching
|
||||
yt-dlp's private write helpers.
|
||||
"""
|
||||
user_opts = self._build_ytdl_options(ytdl_options_presets, ytdl_options_overrides)
|
||||
wants = ('writeinfojson', 'writedescription', 'writethumbnail', 'write_all_thumbnails')
|
||||
if not any(user_opts.get(key) for key in wants):
|
||||
return
|
||||
# An explicit allow_playlist_files=false is the user asking for exactly
|
||||
# this to not happen.
|
||||
if user_opts.get('allow_playlist_files') is False:
|
||||
return
|
||||
|
||||
dldirectory, error_message = self.__calc_download_path(download_type, folder)
|
||||
if error_message is not None:
|
||||
return
|
||||
|
||||
template = (
|
||||
self.config.OUTPUT_TEMPLATE_CHANNEL if etype == 'channel'
|
||||
else self.config.OUTPUT_TEMPLATE_PLAYLIST
|
||||
) or self.config.OUTPUT_TEMPLATE
|
||||
|
||||
debug_logging = logging.getLogger().isEnabledFor(logging.DEBUG)
|
||||
params = {
|
||||
**user_opts,
|
||||
'quiet': not debug_logging,
|
||||
'verbose': debug_logging,
|
||||
'no_color': True,
|
||||
'skip_download': True,
|
||||
'extract_flat': True,
|
||||
'allow_playlist_files': True,
|
||||
'paths': {"home": dldirectory, "temp": self.config.TEMP_DIR},
|
||||
# Feed-level keys only; per-item names are resolved by __add_download.
|
||||
'outtmpl': {
|
||||
'pl_infojson': template,
|
||||
'pl_thumbnail': template,
|
||||
'pl_description': template,
|
||||
},
|
||||
}
|
||||
imp = user_opts.get('impersonate')
|
||||
if imp is not None:
|
||||
params['impersonate'] = yt_dlp.networking.impersonate.ImpersonateTarget.from_str(imp)
|
||||
|
||||
# A copy: process_ie_result mutates entries/requested_entries, and the
|
||||
# caller still needs the real feed dict to queue the items.
|
||||
feed = {k: v for k, v in entry.items() if k != 'entries'}
|
||||
feed['entries'] = []
|
||||
yt_dlp.YoutubeDL(params=params).process_ie_result(feed, download=False)
|
||||
|
||||
async def __write_feed_metadata(self, entry, etype, download_type, folder,
|
||||
ytdl_options_presets, ytdl_options_overrides):
|
||||
try:
|
||||
await asyncio.get_running_loop().run_in_executor(
|
||||
None,
|
||||
partial(
|
||||
self.__write_feed_metadata_sync, entry, etype, download_type, folder,
|
||||
ytdl_options_presets, ytdl_options_overrides,
|
||||
),
|
||||
)
|
||||
except Exception as exc:
|
||||
# Supplemental output must never fail the add.
|
||||
log.warning(f'Could not write {etype} metadata files: {exc}')
|
||||
|
||||
async def __add_entry(
|
||||
self,
|
||||
entry,
|
||||
@@ -1260,6 +1483,7 @@ class DownloadQueue:
|
||||
clip_end,
|
||||
already,
|
||||
_add_gen=None,
|
||||
retry_entry=None,
|
||||
):
|
||||
if not entry:
|
||||
return {'status': 'error', 'msg': "Invalid/empty data was given."}
|
||||
@@ -1278,6 +1502,10 @@ class DownloadQueue:
|
||||
|
||||
if etype.startswith('url'):
|
||||
log.debug('Processing as a url')
|
||||
# retry_entry must ride along: extraction can hand back an
|
||||
# unprocessed url/url_transparent result, and dropping the retry
|
||||
# context here would send the retried item back to the root
|
||||
# directory instead of its original playlist folder.
|
||||
return await self.add(
|
||||
entry['url'],
|
||||
download_type,
|
||||
@@ -1298,6 +1526,7 @@ class DownloadQueue:
|
||||
clip_end,
|
||||
already,
|
||||
_add_gen,
|
||||
retry_entry,
|
||||
)
|
||||
elif etype == 'playlist' or etype == 'channel':
|
||||
if etype == 'playlist' and self.__is_channel_extraction(entry):
|
||||
@@ -1309,6 +1538,10 @@ class DownloadQueue:
|
||||
entries = list(entries)
|
||||
total_entries = len(entries)
|
||||
log.info(f'{etype} detected with {total_entries} entries')
|
||||
await self.__write_feed_metadata(
|
||||
entry, etype, download_type, folder,
|
||||
ytdl_options_presets, ytdl_options_overrides,
|
||||
)
|
||||
index_digits = len(str(total_entries))
|
||||
results = []
|
||||
if playlist_item_limit > 0:
|
||||
@@ -1406,6 +1639,59 @@ class DownloadQueue:
|
||||
return {'status': 'ok'}
|
||||
return {'status': 'error', 'msg': f'Unsupported resource "{etype}"'}
|
||||
|
||||
async def __record_add_failure(
|
||||
self,
|
||||
url,
|
||||
msg,
|
||||
download_type,
|
||||
codec,
|
||||
format,
|
||||
quality,
|
||||
folder,
|
||||
custom_name_prefix,
|
||||
playlist_item_limit,
|
||||
split_by_chapters,
|
||||
chapter_template,
|
||||
subtitle_language,
|
||||
subtitle_mode,
|
||||
ytdl_options_presets,
|
||||
ytdl_options_overrides,
|
||||
clip_start,
|
||||
clip_end,
|
||||
entry=None,
|
||||
):
|
||||
"""Surface a URL that failed before a DownloadInfo could be created (unsupported
|
||||
URL, SSRF-rejected, extraction error) as a failed entry in the done list, so the
|
||||
frontend shows it with the same red-cross/retry/error-detail treatment as a
|
||||
download that failed mid-stream, instead of only a toast and a server log line."""
|
||||
info = DownloadInfo(
|
||||
id=url,
|
||||
title=_short_title_for_failed_url(url),
|
||||
url=url,
|
||||
quality=quality,
|
||||
download_type=download_type,
|
||||
codec=codec,
|
||||
format=format,
|
||||
folder=folder,
|
||||
custom_name_prefix=custom_name_prefix,
|
||||
error=msg,
|
||||
entry=entry,
|
||||
playlist_item_limit=playlist_item_limit,
|
||||
split_by_chapters=split_by_chapters,
|
||||
chapter_template=chapter_template,
|
||||
subtitle_language=subtitle_language,
|
||||
subtitle_mode=subtitle_mode,
|
||||
ytdl_options_presets=ytdl_options_presets,
|
||||
ytdl_options_overrides=ytdl_options_overrides,
|
||||
clip_start=clip_start,
|
||||
clip_end=clip_end,
|
||||
)
|
||||
info.status = 'error'
|
||||
info.msg = msg
|
||||
download = Download(None, None, None, None, quality, format, {}, info)
|
||||
self.done.put(download)
|
||||
await self.notifier.completed(info)
|
||||
|
||||
async def add(
|
||||
self,
|
||||
url,
|
||||
@@ -1427,6 +1713,7 @@ class DownloadQueue:
|
||||
clip_end=None,
|
||||
already=None,
|
||||
_add_gen=None,
|
||||
retry_entry=None,
|
||||
):
|
||||
if ytdl_options_presets is None:
|
||||
ytdl_options_presets = []
|
||||
@@ -1444,13 +1731,37 @@ class DownloadQueue:
|
||||
return {'status': 'ok'}
|
||||
else:
|
||||
already.add(url)
|
||||
# SSRF guard: reject non-http(s) schemes and hosts resolving to
|
||||
# internal/loopback/link-local/metadata addresses before yt-dlp fetches
|
||||
# anything. run_in_executor because validate_url may perform a DNS lookup.
|
||||
url_error = await asyncio.get_running_loop().run_in_executor(
|
||||
None, partial(validate_url, url, allow_private=self.config.ALLOW_PRIVATE_ADDRESSES))
|
||||
if url_error is not None:
|
||||
log.warning('Rejected URL "%s": %s', url, url_error)
|
||||
await self.__record_add_failure(
|
||||
url, url_error, download_type, codec, format, quality, folder,
|
||||
custom_name_prefix, playlist_item_limit, split_by_chapters, chapter_template,
|
||||
subtitle_language, subtitle_mode, ytdl_options_presets, ytdl_options_overrides,
|
||||
clip_start, clip_end, retry_entry,
|
||||
)
|
||||
return {'status': 'error', 'msg': url_error}
|
||||
try:
|
||||
entry = await asyncio.get_running_loop().run_in_executor(
|
||||
None,
|
||||
partial(self.__extract_info, url, ytdl_options_presets, ytdl_options_overrides),
|
||||
)
|
||||
except yt_dlp.utils.YoutubeDLError as exc:
|
||||
return {'status': 'error', 'msg': str(exc)}
|
||||
msg = str(exc)
|
||||
await self.__record_add_failure(
|
||||
url, msg, download_type, codec, format, quality, folder,
|
||||
custom_name_prefix, playlist_item_limit, split_by_chapters, chapter_template,
|
||||
subtitle_language, subtitle_mode, ytdl_options_presets, ytdl_options_overrides,
|
||||
clip_start, clip_end, retry_entry,
|
||||
)
|
||||
return {'status': 'error', 'msg': msg}
|
||||
retry_context = _compact_persisted_entry(retry_entry)
|
||||
if isinstance(entry, dict) and retry_context is not None:
|
||||
entry = {**entry, **copy.deepcopy(retry_context)}
|
||||
return await self.__add_entry(
|
||||
entry,
|
||||
download_type,
|
||||
@@ -1471,6 +1782,43 @@ class DownloadQueue:
|
||||
clip_end,
|
||||
already,
|
||||
_add_gen,
|
||||
retry_entry,
|
||||
)
|
||||
|
||||
async def retry(self, id):
|
||||
if not self.done.exists(id):
|
||||
return {'status': 'error', 'msg': 'Failed download no longer exists.'}
|
||||
|
||||
info = self.done.get(id).info
|
||||
if info.status != 'error':
|
||||
return {'status': 'error', 'msg': 'Only failed downloads can be retried.'}
|
||||
|
||||
# The stored options were validated by parse_download_options when the
|
||||
# download was first submitted, but the configuration can have changed
|
||||
# since. Re-apply the same gates here so a retry can't resurrect
|
||||
# overrides or presets the current configuration no longer allows.
|
||||
overrides = info.ytdl_options_overrides if self.config.ALLOW_YTDL_OPTIONS_OVERRIDES else {}
|
||||
presets = [p for p in info.ytdl_options_presets if p in self.config.YTDL_OPTIONS_PRESETS]
|
||||
|
||||
return await self.add(
|
||||
info.url,
|
||||
info.download_type,
|
||||
info.codec,
|
||||
info.format,
|
||||
info.quality,
|
||||
info.folder,
|
||||
info.custom_name_prefix,
|
||||
info.playlist_item_limit,
|
||||
True,
|
||||
info.split_by_chapters,
|
||||
info.chapter_template,
|
||||
info.subtitle_language,
|
||||
info.subtitle_mode,
|
||||
presets,
|
||||
overrides,
|
||||
info.clip_start,
|
||||
info.clip_end,
|
||||
retry_entry=info.entry,
|
||||
)
|
||||
|
||||
async def add_entry(
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 885 KiB After Width: | Height: | Size: 1.9 MiB |
+8
-8
@@ -33,10 +33,10 @@
|
||||
"@angular/platform-browser-dynamic": "^22.0.6",
|
||||
"@angular/service-worker": "^22.0.6",
|
||||
"@fortawesome/angular-fontawesome": "~4.0.0",
|
||||
"@fortawesome/fontawesome-svg-core": "^7.3.0",
|
||||
"@fortawesome/free-brands-svg-icons": "^7.3.0",
|
||||
"@fortawesome/free-regular-svg-icons": "^7.3.0",
|
||||
"@fortawesome/free-solid-svg-icons": "^7.3.0",
|
||||
"@fortawesome/fontawesome-svg-core": "^7.3.1",
|
||||
"@fortawesome/free-brands-svg-icons": "^7.3.1",
|
||||
"@fortawesome/free-regular-svg-icons": "^7.3.1",
|
||||
"@fortawesome/free-solid-svg-icons": "^7.3.1",
|
||||
"@ng-bootstrap/ng-bootstrap": "^21.0.0",
|
||||
"@ng-select/ng-select": "^23.2.0",
|
||||
"@popperjs/core": "^2.11.8",
|
||||
@@ -49,13 +49,13 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@angular-eslint/builder": "22.0.0",
|
||||
"@angular/build": "^22.0.5",
|
||||
"@angular/cli": "^22.0.5",
|
||||
"@angular/build": "^22.0.7",
|
||||
"@angular/cli": "^22.0.7",
|
||||
"@angular/compiler-cli": "^22.0.6",
|
||||
"@angular/localize": "^22.0.6",
|
||||
"@eslint/js": "^9.39.4",
|
||||
"@eslint/js": "^9.39.5",
|
||||
"angular-eslint": "22.0.0",
|
||||
"eslint": "^9.39.4",
|
||||
"eslint": "^9.39.5",
|
||||
"jsdom": "^27.4.0",
|
||||
"typescript": "~6.0.3",
|
||||
"typescript-eslint": "8.62.0",
|
||||
|
||||
Generated
+258
-285
File diff suppressed because it is too large
Load Diff
+27
-1
@@ -958,7 +958,33 @@
|
||||
[disabled]="downloads.loading"
|
||||
[attr.aria-label]="'Select subscription ' + entry[1].name" />
|
||||
</td>
|
||||
<td>{{ entry[1].name }}</td>
|
||||
<td>
|
||||
@if (editingNameId === entry[0]) {
|
||||
<div class="d-flex flex-wrap gap-1 align-items-center">
|
||||
<input type="text"
|
||||
class="form-control form-control-sm flex-grow-1"
|
||||
[name]="'subName' + entry[0]"
|
||||
[(ngModel)]="nameEditDraft"
|
||||
[maxlength]="subscriptionNameMaxLength"
|
||||
[disabled]="downloads.loading"
|
||||
[attr.aria-label]="'Subscription name for ' + entry[1].name" />
|
||||
<button type="button" class="btn btn-sm btn-outline-secondary"
|
||||
(click)="saveName(entry[0])"
|
||||
[disabled]="downloads.loading">Save</button>
|
||||
<button type="button" class="btn btn-sm btn-outline-secondary"
|
||||
(click)="cancelEditName()"
|
||||
[disabled]="downloads.loading">Cancel</button>
|
||||
</div>
|
||||
} @else {
|
||||
<div class="d-flex flex-wrap gap-1 align-items-center">
|
||||
<span class="text-break">{{ entry[1].name }}</span>
|
||||
<button type="button" class="btn btn-link btn-sm p-0"
|
||||
(click)="beginEditName(entry[0], entry[1].name)"
|
||||
[disabled]="downloads.loading"
|
||||
ngbTooltip="Rename this subscription (display name only; does not affect the download folder)">Edit</button>
|
||||
</div>
|
||||
}
|
||||
</td>
|
||||
<td class="text-break"><a [href]="entry[1].url" target="_blank" rel="noopener">{{ entry[1].url }}</a></td>
|
||||
<td>
|
||||
@if (editingTitleRegexId === entry[0]) {
|
||||
|
||||
+70
-1
@@ -19,6 +19,7 @@ class DownloadsServiceStub {
|
||||
customDirsChanged = new Subject<Record<string, string[]>>();
|
||||
ytdlOptionsChanged = new Subject<Record<string, unknown>>();
|
||||
updated = new Subject<void>();
|
||||
retryCalls: string[] = [];
|
||||
|
||||
getCookieStatus() {
|
||||
return of({ status: 'ok', has_cookies: false });
|
||||
@@ -32,6 +33,11 @@ class DownloadsServiceStub {
|
||||
return of({ status: 'ok' as const });
|
||||
}
|
||||
|
||||
retry(id: string) {
|
||||
this.retryCalls.push(id);
|
||||
return of({ status: 'ok' as const });
|
||||
}
|
||||
|
||||
cancelAdd() {
|
||||
return of({ status: 'ok' as const });
|
||||
}
|
||||
@@ -75,7 +81,10 @@ class SubscriptionsServiceStub {
|
||||
return of({});
|
||||
}
|
||||
|
||||
update() {
|
||||
updateCalls: [string, unknown][] = [];
|
||||
|
||||
update(id: string, changes: unknown) {
|
||||
this.updateCalls.push([id, changes]);
|
||||
return of({ status: 'ok' as const });
|
||||
}
|
||||
|
||||
@@ -269,6 +278,33 @@ describe('App', () => {
|
||||
expect(payload.clipEnd).toBe('1:20');
|
||||
});
|
||||
|
||||
it('retries a failed download by its server-side queue id', () => {
|
||||
const fixture = TestBed.createComponent(App);
|
||||
const app = fixture.componentInstance;
|
||||
const download = {
|
||||
id: 'vid1',
|
||||
title: 'Test Video',
|
||||
url: 'https://example.com/v',
|
||||
download_type: 'video',
|
||||
quality: 'best',
|
||||
format: 'any',
|
||||
folder: '',
|
||||
custom_name_prefix: '',
|
||||
playlist_item_limit: 0,
|
||||
status: 'error',
|
||||
msg: 'temporary failure',
|
||||
percent: 0,
|
||||
speed: 0,
|
||||
eta: 0,
|
||||
filename: '',
|
||||
checked: false,
|
||||
};
|
||||
|
||||
app.retryDownload(download.url, download);
|
||||
|
||||
expect(downloads.retryCalls).toEqual([download.url]);
|
||||
});
|
||||
|
||||
it('blocks subscribe with invalid title regex', () => {
|
||||
const toasts = TestBed.inject(ToastService);
|
||||
const errorSpy = vi.spyOn(toasts, 'error').mockImplementation(() => undefined);
|
||||
@@ -282,4 +318,37 @@ describe('App', () => {
|
||||
expect(errorSpy).toHaveBeenCalledWith('Invalid subscription title filter (regex)');
|
||||
errorSpy.mockRestore();
|
||||
});
|
||||
|
||||
it('renames a subscription and closes the inline editor', () => {
|
||||
const fixture = TestBed.createComponent(App);
|
||||
const app = fixture.componentInstance;
|
||||
const subs = TestBed.inject(SubscriptionsService) as unknown as SubscriptionsServiceStub;
|
||||
|
||||
app.beginEditName('sub1', 'Videos');
|
||||
expect(app.editingNameId).toBe('sub1');
|
||||
expect(app.nameEditDraft).toBe('Videos');
|
||||
|
||||
app.nameEditDraft = ' Jane uploads ';
|
||||
app.saveName('sub1');
|
||||
|
||||
expect(subs.updateCalls).toEqual([['sub1', { name: 'Jane uploads' }]]);
|
||||
expect(app.editingNameId).toBeNull();
|
||||
});
|
||||
|
||||
it('blocks renaming a subscription to an empty name', () => {
|
||||
const toasts = TestBed.inject(ToastService);
|
||||
const errorSpy = vi.spyOn(toasts, 'error').mockImplementation(() => undefined);
|
||||
const fixture = TestBed.createComponent(App);
|
||||
const app = fixture.componentInstance;
|
||||
const subs = TestBed.inject(SubscriptionsService) as unknown as SubscriptionsServiceStub;
|
||||
|
||||
app.beginEditName('sub1', 'Videos');
|
||||
app.nameEditDraft = ' ';
|
||||
app.saveName('sub1');
|
||||
|
||||
expect(subs.updateCalls.length).toBe(0);
|
||||
expect(app.editingNameId).toBe('sub1');
|
||||
expect(errorSpy).toHaveBeenCalledWith('Subscription name must not be empty');
|
||||
errorSpy.mockRestore();
|
||||
});
|
||||
});
|
||||
|
||||
+32
-22
@@ -102,6 +102,9 @@ export class App implements AfterViewInit, OnInit, OnDestroy {
|
||||
skipSubscriberOnly = false;
|
||||
editingTitleRegexId: string | null = null;
|
||||
titleRegexEditDraft = '';
|
||||
editingNameId: string | null = null;
|
||||
nameEditDraft = '';
|
||||
readonly subscriptionNameMaxLength = 200;
|
||||
cachedSubs: [string, SubscriptionRow][] = [];
|
||||
selectedSubscriptionIds = new Set<string>();
|
||||
checkingSubscriptionIds = new Set<string>();
|
||||
@@ -663,6 +666,34 @@ export class App implements AfterViewInit, OnInit, OnDestroy {
|
||||
});
|
||||
}
|
||||
|
||||
beginEditName(id: string, current: string | undefined) {
|
||||
this.editingNameId = id;
|
||||
this.nameEditDraft = current ?? '';
|
||||
this.cdr.markForCheck();
|
||||
}
|
||||
|
||||
cancelEditName() {
|
||||
this.editingNameId = null;
|
||||
this.nameEditDraft = '';
|
||||
this.cdr.markForCheck();
|
||||
}
|
||||
|
||||
saveName(id: string) {
|
||||
const name = (this.nameEditDraft || '').trim();
|
||||
if (!name) {
|
||||
this.toasts.error('Subscription name must not be empty');
|
||||
return;
|
||||
}
|
||||
this.subscriptionsSvc.update(id, { name }).subscribe((res) => {
|
||||
const error = this.getStatusError(res);
|
||||
if (error) {
|
||||
this.toasts.error(error || 'Update subscription failed');
|
||||
return;
|
||||
}
|
||||
this.cancelEditName();
|
||||
});
|
||||
}
|
||||
|
||||
deleteSubscription(id: string) {
|
||||
this.subscriptionsSvc.delete([id]).subscribe((res) => {
|
||||
const error = this.getStatusError(res);
|
||||
@@ -1146,30 +1177,9 @@ export class App implements AfterViewInit, OnInit, OnDestroy {
|
||||
}
|
||||
|
||||
retryDownload(key: string, download: Download) {
|
||||
const payload = this.buildAddPayload({
|
||||
url: download.url,
|
||||
downloadType: download.download_type,
|
||||
codec: download.codec,
|
||||
quality: download.quality,
|
||||
format: download.format,
|
||||
folder: download.folder,
|
||||
customNamePrefix: download.custom_name_prefix,
|
||||
playlistItemLimit: download.playlist_item_limit,
|
||||
autoStart: true,
|
||||
splitByChapters: download.split_by_chapters,
|
||||
chapterTemplate: download.chapter_template,
|
||||
subtitleLanguage: download.subtitle_language,
|
||||
subtitleMode: download.subtitle_mode,
|
||||
ytdlOptionsPresets: download.ytdl_options_presets?.length
|
||||
? [...download.ytdl_options_presets]
|
||||
: [],
|
||||
ytdlOptionsOverrides: download.ytdl_options_overrides ? JSON.stringify(download.ytdl_options_overrides) : '',
|
||||
clipStart: download.clip_start != null ? String(download.clip_start) : '',
|
||||
clipEnd: download.clip_end != null ? String(download.clip_end) : '',
|
||||
});
|
||||
// Only remove the done-list record once the retry is confirmed queued —
|
||||
// deleting it eagerly would silently lose history if the re-add fails.
|
||||
this.downloads.add(payload)
|
||||
this.downloads.retry(key)
|
||||
.pipe(takeUntilDestroyed(this.destroyRef))
|
||||
.subscribe((status: Status) => {
|
||||
if (status.status === 'error') {
|
||||
|
||||
@@ -117,6 +117,14 @@ describe('DownloadsService', () => {
|
||||
req.flush({ presets: ['Preset A'] });
|
||||
});
|
||||
|
||||
it('retry() posts the failed download id', () => {
|
||||
service.retry('https://example.com/v').subscribe();
|
||||
const req = httpMock.expectOne('retry');
|
||||
expect(req.request.method).toBe('POST');
|
||||
expect(req.request.body).toEqual({ id: 'https://example.com/v' });
|
||||
req.flush({ status: 'ok' });
|
||||
});
|
||||
|
||||
it('cancelAdd posts to cancel-add', () => {
|
||||
service.cancelAdd().subscribe();
|
||||
const req = httpMock.expectOne('cancel-add');
|
||||
|
||||
@@ -169,6 +169,12 @@ export class DownloadsService {
|
||||
);
|
||||
}
|
||||
|
||||
public retry(id: string) {
|
||||
return this.http.post<Status>('retry', { id: id }).pipe(
|
||||
catchError(this.handleHTTPError)
|
||||
);
|
||||
}
|
||||
|
||||
public startById(ids: string[]) {
|
||||
return this.http.post<Status>('start', {ids: ids}).pipe(
|
||||
catchError(this.handleHTTPError)
|
||||
|
||||
@@ -106,14 +106,14 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "anyio"
|
||||
version = "4.14.1"
|
||||
version = "4.14.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "idna" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/3b/72/5562aabb8dd7181e8e860622a38bea08d17842b99ecd4c91f84ac95251b0/anyio-4.14.1.tar.gz", hash = "sha256:8d648a3544c1a700e3ff78615cd679e4c5c3f149904287e73687b2596963629e", size = 254831, upload-time = "2026-06-24T20:56:06.017Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/61/cc/a381afa6efea9f496eff839d4a6a1aed3bfafc7b3ab4b0d1b243a12573dd/anyio-4.14.2.tar.gz", hash = "sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f", size = 260176, upload-time = "2026-07-12T20:29:07.082Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b0/7b/90df4a0a816d98d6ea26f559d87836d494a2cf1fcf063be67df50a7bcc30/anyio-4.14.1-py3-none-any.whl", hash = "sha256:4e5533c5b8ff0a24f5d7a176cbe6877129cd183893f66b537f8f227d10527d72", size = 124875, upload-time = "2026-06-24T20:56:04.413Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -366,15 +366,15 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "deno"
|
||||
version = "2.9.2"
|
||||
version = "2.9.3"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/dc/3c/bf3a4e2c00f7eefd4ed2fd033a958cab3e92dcc539d3ab80351850cae99c/deno-2.9.2.tar.gz", hash = "sha256:3b77d7689c15fb2c47d5a3927dc8cf70dbd408bc92e20c70da079c6d78980e34", size = 8164, upload-time = "2026-07-08T14:38:54.267Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/98/ab/638749d76881f74d100a079414745f0a0fd20bba38e27ab4a3a6495d8264/deno-2.9.3.tar.gz", hash = "sha256:73268cbac7f7c4ff1983e49420a93f3f3f2f2e4e4372436f0942d04d02c9e943", size = 8166, upload-time = "2026-07-15T15:37:50.565Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/27/fe/4a02a256f19b3f31ac17ec37f6467ecf523601ed753e6c9177865dd2bbb2/deno-2.9.2-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:fb4bb5362d02c193f8086668789076f366416980b79277c06ea6cff73866b668", size = 42341183, upload-time = "2026-07-08T14:38:36.384Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5a/3e/235530ac9b9c206ee8c098e2b215f711a71d915d8d7d2be77599c8c1db4b/deno-2.9.2-py3-none-macosx_11_0_arm64.whl", hash = "sha256:f6f74fd8698a8a9a31af14eed5730f1964ab50eaa0129079d11d91fa92496f60", size = 37984508, upload-time = "2026-07-08T14:38:40.006Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/7b/27/06a166fca538c06dfae4c45dbb92c2ebf3083341141966a3d649272ea734/deno-2.9.2-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:8b2fd9318d452b7e44c45968916670a0c5452d72796e1af9abfaa7a389e4aa6b", size = 42089854, upload-time = "2026-07-08T14:38:43.841Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/09/b4/38338550a5ca0a7bcc39991e3d48e46ce3699413460dff32f383a611d822/deno-2.9.2-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:bf0471db7238e90810bca77a8d2c28646317b5a3fd9e5cdfdf73b251fde11b83", size = 43916921, upload-time = "2026-07-08T14:38:48.057Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a9/3d/c490c27e1f720baf014bb2c18b29d09b11c21d69fadfd69639e5904cdc95/deno-2.9.2-py3-none-win_amd64.whl", hash = "sha256:8e815d66b3e1314d028b2f0a05c75a331dd7025f4c825e12dd1f057207cee1f6", size = 41621982, upload-time = "2026-07-08T14:38:51.928Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/07/fa/b4e1e2b3b894ed992c78a6796a43dc775edff895239d83c0e9a10457f4f7/deno-2.9.3-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:1deef9fe97ab28d5d4fea07693075e9ccc62f4ea97a653047a770beb01ecd307", size = 42354026, upload-time = "2026-07-15T15:37:34.977Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/3d/cc/4090ef7370005b740efe57ec6ff75a3471393abe9816f1459e73dbbf0429/deno-2.9.3-py3-none-macosx_11_0_arm64.whl", hash = "sha256:78635cd9f0802ce7125aa3ba6feb5775460f55d432611374f16b454ffb8a1308", size = 37997909, upload-time = "2026-07-15T15:37:38.291Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/14/07/70b7c965bbca5f3d158859acda0e90c041f1fa98b8c9ac8eff3fafd8b96c/deno-2.9.3-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:7c1c669827a480ef9e2710dd762560811da35dca04a9e90b23c0518587dadb8a", size = 42100873, upload-time = "2026-07-15T15:37:41.539Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/7b/f5/0a07cc19a27476011719dd017a98950c7ccc33058c2ffc07019ba3ad9a5e/deno-2.9.3-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:3e724d5c8df13a90f6b50b16cc8df1acfe64c61a7736a774e11d61f3bca3a3b1", size = 43928967, upload-time = "2026-07-15T15:37:45.175Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e1/9a/030847bd4ea6cefbb471cae2b63c9ab29a3ab6c1809484447752bbcbcd52/deno-2.9.3-py3-none-win_amd64.whl", hash = "sha256:e2ebe2b5c1a7ee9daeaf8caf14ebc33fda96b4c2399da259e761fe2f82d22fa4", size = 41630201, upload-time = "2026-07-15T15:37:48.344Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
Reference in New Issue
Block a user