mirror of
https://github.com/alexta69/metube.git
synced 2026-07-23 13:22:48 +00:00
Compare commits
14 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 1c7261ab59 | |||
| a23d1689e3 | |||
| 4b05022b91 | |||
| 0dc9b0b3d6 | |||
| 96f5ffe34a | |||
| 1eafecd1cc | |||
| 8fe81dea18 | |||
| f054d84108 | |||
| fa02717e12 | |||
| 9ca78be199 | |||
| 8071611a84 | |||
| 220f991fae | |||
| 6d0528783c | |||
| c104e30451 |
@@ -14,7 +14,7 @@ jobs:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v7
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v6
|
||||
uses: actions/setup-node@v7
|
||||
with:
|
||||
node-version: lts/*
|
||||
- name: Enable pnpm
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
name: readme-size
|
||||
|
||||
on:
|
||||
push:
|
||||
paths:
|
||||
- 'README.md'
|
||||
pull_request:
|
||||
paths:
|
||||
- 'README.md'
|
||||
|
||||
jobs:
|
||||
check-size:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v7
|
||||
- name: Check README stays under Docker Hub's 25k character limit
|
||||
run: |
|
||||
size=$(wc -c < README.md)
|
||||
echo "README.md is ${size} bytes (limit 25000)"
|
||||
test "$size" -lt 25000
|
||||
@@ -1,5 +1,53 @@
|
||||
# Agent Guidelines
|
||||
|
||||
## Project scope — read this before planning a feature
|
||||
|
||||
MeTube's contract is: give it a URL, it runs yt-dlp well, and correct files appear.
|
||||
The maintainer holds a deliberate line on what belongs inside that contract, and PRs
|
||||
on the wrong side of it are declined **regardless of code quality**. Check your plan
|
||||
against this line before writing any code.
|
||||
|
||||
**In scope — improving the write itself:**
|
||||
|
||||
- Features that make the file yt-dlp writes at download time come out more correct,
|
||||
using only data the extractor already provides (e.g. filling a missing album-artist
|
||||
tag from the extractor's own metadata).
|
||||
- Surfacing functionality yt-dlp itself owns and maintains as first-class UI options
|
||||
(e.g. a SponsorBlock toggle that just passes yt-dlp postprocessor params).
|
||||
- Download queue, subscriptions, output templates, and UI improvements to the
|
||||
download workflow.
|
||||
|
||||
**Out of scope — managing files after they exist:**
|
||||
|
||||
- Tag editors, metadata dialogs, or any workflow that rewrites files after the
|
||||
download has finished. This holds even for slimmed-down versions.
|
||||
- Lookups against external metadata services (iTunes, Deezer, MusicBrainz, etc.).
|
||||
More broadly: any new dependency on a third-party API, or new network egress from
|
||||
self-hosted instances, beyond what yt-dlp itself performs.
|
||||
- Library organization: moving/renaming existing files into Artist/Album layouts,
|
||||
watch-folder processing, and similar media-manager features. Dedicated tools
|
||||
(beets, MusicBrainz Picard, Lidarr) do this properly; the README points users
|
||||
to them.
|
||||
|
||||
**Corollaries that shape borderline PRs:**
|
||||
|
||||
- Site-specific intelligence (parsing playlist-ID prefixes, URL path conventions,
|
||||
and other platform internals) is extractor work and belongs upstream in yt-dlp,
|
||||
not re-implemented here — it silently breaks when the platform changes and
|
||||
MeTube would own the breakage.
|
||||
- Prefer enriching yt-dlp's info dict and letting its existing pipeline
|
||||
(FFmpegMetadata etc.) do the writing, over adding custom per-format tag-writing
|
||||
code to MeTube.
|
||||
- Supplemental processing must never fail a download that otherwise succeeded:
|
||||
warn and continue, don't raise.
|
||||
- Keep feature scope minimal on first submission. A hardcoded sensible default
|
||||
beats a configuration surface; follow-ups can add options when users actually
|
||||
ask. PRs that bundle several "reasonable next steps" invite rejection of the
|
||||
whole.
|
||||
|
||||
If a feature idea fails this test, the accepted alternative is usually a README
|
||||
section documenting how to pair MeTube with the right dedicated tool.
|
||||
|
||||
## README.md size constraint
|
||||
|
||||
The README.md is synced to Docker Hub, which has a **25,000 character limit**.
|
||||
@@ -30,6 +78,24 @@ uv run pytest app/tests/
|
||||
|
||||
All of these run in CI (`.github/workflows/main.yml`) on every push to master and must pass.
|
||||
|
||||
Gotchas:
|
||||
|
||||
- Backend tests must run **from the repo root**: `main.py` resolves the static-assets
|
||||
path relative to the cwd, and several test modules import `main`. Running from
|
||||
`app/` makes five test modules fail to import.
|
||||
- The frontend must be **built before** running backend tests (same reason — the
|
||||
assets at `ui/dist/metube/browser` must exist). The command order above is
|
||||
load-bearing.
|
||||
- `app/tests/test_ytdl_utils.py` stubs `yt_dlp` at import time. Run standalone,
|
||||
two tests fail with `AttributeError: <module 'yt_dlp'> does not have the
|
||||
attribute 'YoutubeDL'`; under the full suite the real module is imported first
|
||||
and they pass. This is a known quirk, not a bug to fix in the code under test.
|
||||
|
||||
Every non-markdown push to master builds multi-arch Docker images and cuts a dated
|
||||
release the same day. **Master is continuously released** — a PR must be
|
||||
release-ready exactly as merged; there is no stabilization window for follow-up
|
||||
fixes.
|
||||
|
||||
## Code style
|
||||
|
||||
Follow `.editorconfig`:
|
||||
@@ -56,5 +122,36 @@ ui/src/app/ — Angular standalone components (no NgModules)
|
||||
- Backend configuration lives in the `Config` class in `app/main.py` with env-var defaults in `_DEFAULTS`. New env vars go there.
|
||||
- Real-time communication uses Socket.IO events, not REST polling.
|
||||
- Frontend uses standalone Angular components with `inject()` for DI, RxJS Subjects for state, and `takeUntilDestroyed()` for cleanup.
|
||||
- Frontend components use OnPush change detection: subscribe callbacks must call `cdr.markForCheck()`.
|
||||
- State is persisted as JSON files via `AtomicJsonStore` in `app/state_store.py`.
|
||||
- Persisted state stays compact: the completed queue deliberately drops bulky entry data (see `_compact_persisted_entry` in `app/ytdl.py`). Don't expand what gets persisted without discussion.
|
||||
- Custom yt-dlp postprocessors added to `ytdl_params['postprocessors']` run in **list order** within a stage. When combining postprocessors, mirror the ordering the yt-dlp CLI would produce (e.g. sponsor-segment removal before chapter splitting).
|
||||
- No pre-commit hooks — linting and tests are enforced in CI only.
|
||||
|
||||
## Checklist: adding a per-download option
|
||||
|
||||
New options on the download form (the `split_by_chapters` pattern) need **all** of
|
||||
these pieces — the last three are the ones commonly missed:
|
||||
|
||||
1. `parse_download_options` in `app/main.py`.
|
||||
2. A field on `DownloadInfo` in `app/ytdl.py`.
|
||||
3. A `hasattr` backfill in `DownloadInfo.__setstate__` for old persisted records.
|
||||
4. The safe-deserialization field list in `app/ytdl.py`.
|
||||
5. UI form control + cookie persistence in `ui/src/app/app.ts` / `app.html`, and
|
||||
the payload in `downloads.service.ts` (plus its spec).
|
||||
6. The redownload path in `app.ts`, so retries carry the option.
|
||||
7. If the option makes sense for unattended downloads: threading through
|
||||
`app/subscriptions.py` (`SubscriptionInfo` field, serializer, add/update
|
||||
routes, the enqueue call) — or a note in the PR that it's deliberately
|
||||
direct-downloads-only.
|
||||
|
||||
## Security invariants
|
||||
|
||||
User input and extractor-provided metadata (titles, playlist names, URLs) are
|
||||
untrusted. Use the existing guards instead of hand-rolling:
|
||||
|
||||
- User-submitted URLs go through the SSRF guard (see `test_url_guard.py` for the
|
||||
expected behavior).
|
||||
- Anything that becomes a filesystem path goes through `_is_within_directory` and
|
||||
`_sanitize_path_component` in `app/ytdl.py` — including values that arrive via
|
||||
yt-dlp metadata, which sites can influence.
|
||||
|
||||
@@ -3,12 +3,12 @@
|
||||

|
||||

|
||||
|
||||
MeTube is a self-hosted web UI for `yt-dlp`, for downloading media from YouTube and [dozens of other sites](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md).
|
||||
MeTube is a self-hosted web UI for `yt-dlp`, for downloading media from YouTube and [dozens of other sites](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md). Docker images are multi-arch (amd64/arm64).
|
||||
|
||||
Key capabilities:
|
||||
* Download videos, audio, captions, and thumbnails from a browser UI.
|
||||
* Download playlists and channels, with configurable output and download options.
|
||||
* Subscribe to channels and playlists, periodically check for new items, and queue new uploads automatically.
|
||||
* [Subscribe](https://github.com/alexta69/metube/wiki/Subscriptions) to channels and playlists, periodically check for new items, and queue new uploads automatically.
|
||||
|
||||

|
||||
|
||||
@@ -18,7 +18,7 @@ Key capabilities:
|
||||
docker run -d -p 8081:8081 -v /path/to/downloads:/downloads ghcr.io/alexta69/metube
|
||||
```
|
||||
|
||||
## 🐳 Run using docker-compose
|
||||
## 🐳 Run using Docker Compose
|
||||
|
||||
```yaml
|
||||
services:
|
||||
@@ -34,7 +34,16 @@ services:
|
||||
|
||||
## ⚙️ Configuration via environment variables
|
||||
|
||||
Certain values can be set via environment variables, using the `-e` parameter on the docker command line, or the `environment:` section in docker-compose.
|
||||
Certain values can be set via environment variables, using the `-e` parameter on the docker command line, or the `environment:` section in Docker Compose.
|
||||
|
||||
### 🏠 Runtime & Permissions
|
||||
|
||||
* __PUID__: User under which MeTube will run. Defaults to `1000` (legacy `UID` also supported).
|
||||
* __PGID__: Group under which MeTube will run. Defaults to `1000` (legacy `GID` also supported).
|
||||
* __UMASK__: Umask value used by MeTube. Defaults to `022`.
|
||||
* __DEFAULT_THEME__: Default theme to use for the UI, can be set to `light`, `dark`, or `auto`. Defaults to `auto`.
|
||||
* __LOGLEVEL__: Log level, can be set to `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, or `NONE`. Defaults to `INFO`.
|
||||
* __ENABLE_ACCESSLOG__: Whether to enable access log. Defaults to `false`.
|
||||
|
||||
### ⬇️ Download Behavior
|
||||
|
||||
@@ -83,18 +92,9 @@ Certain values can be set via environment variables, using the `-e` parameter on
|
||||
* __HTTPS__: Use `https` instead of `http` (__CERTFILE__ and __KEYFILE__ required). Defaults to `false`.
|
||||
* __CERTFILE__: HTTPS certificate file path.
|
||||
* __KEYFILE__: HTTPS key file path.
|
||||
* __CORS_ALLOWED_ORIGINS__: Comma-separated list of origins permitted to make cross-origin requests to the MeTube API. When unset or empty, all cross-origin requests are denied. Set to `*` to allow all origins. This must be configured for [browser extensions](#-browser-extensions), [bookmarklets](#-bookmarklet), and any other browser-based tools that contact MeTube from a different origin. For browser extensions use `*` (see below); for bookmarklets you can list specific sites, e.g. `https://www.youtube.com,https://www.vimeo.com`.
|
||||
* __CORS_ALLOWED_ORIGINS__: Comma-separated list of origins permitted to make cross-origin requests to the MeTube API; `*` allows all. When unset or empty, all cross-origin requests are denied. Required for browser extensions and bookmarklets — see [Sending links to MeTube](#-sending-links-to-metube).
|
||||
* __ROBOTS_TXT__: A path to a `robots.txt` file mounted in the container.
|
||||
|
||||
### 🏠 Basic Setup
|
||||
|
||||
* __PUID__: User under which MeTube will run. Defaults to `1000` (legacy `UID` also supported).
|
||||
* __PGID__: Group under which MeTube will run. Defaults to `1000` (legacy `GID` also supported).
|
||||
* __UMASK__: Umask value used by MeTube. Defaults to `022`.
|
||||
* __DEFAULT_THEME__: Default theme to use for the UI, can be set to `light`, `dark`, or `auto`. Defaults to `auto`.
|
||||
* __LOGLEVEL__: Log level, can be set to `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, or `NONE`. Defaults to `INFO`.
|
||||
* __ENABLE_ACCESSLOG__: Whether to enable access log. Defaults to `false`.
|
||||
|
||||
## 🎛️ Configuring yt-dlp options
|
||||
|
||||
MeTube lets you customize how [yt-dlp](https://github.com/yt-dlp/yt-dlp) behaves at three levels, from broadest to most specific:
|
||||
@@ -229,47 +229,27 @@ In case you need to use your browser's cookies with MeTube, for example to downl
|
||||
* After upload, the cookie indicator should show as active.
|
||||
* Use **Delete Cookies** in the same section to remove uploaded cookies.
|
||||
|
||||
## 🔌 Browser extensions
|
||||
## 🔗 Sending links to MeTube
|
||||
|
||||
Browser extensions allow right-clicking videos and sending them directly to MeTube. If you're on an HTTPS page, your MeTube instance must be behind an HTTPS reverse proxy (see below) for extensions to work.
|
||||
Several integrations let you send URLs to MeTube from wherever you are, instead of pasting them into the UI. The browser-based ones make cross-origin requests, so they require `CORS_ALLOWED_ORIGINS` to be set; and if you're on an HTTPS page, your MeTube instance must be served over HTTPS too (with `HTTPS=true` or behind an HTTPS reverse proxy — see below).
|
||||
|
||||
Since browser extensions make requests from their own origin (`chrome-extension://...` or `moz-extension://...`), you must set `CORS_ALLOWED_ORIGINS=*` for them to work.
|
||||
__Browser extensions__ allow right-clicking videos and sending them directly to MeTube. Since extensions request from their own origin, set `CORS_ALLOWED_ORIGINS=*`.
|
||||
* __Chrome:__ contributed by [Rpsl](https://github.com/rpsl) — install from the [Chrome Webstore](https://chrome.google.com/webstore/detail/metube-downloader/fbmkmdnlhacefjljljlbhkodfmfkijdh) or [from sources](https://github.com/Rpsl/metube-browser-extension).
|
||||
* __Firefox:__ contributed by [nanocortex](https://github.com/nanocortex) — install from [Firefox Addons](https://addons.mozilla.org/en-US/firefox/addon/metube-downloader) or get sources [here](https://github.com/nanocortex/metube-firefox-addon).
|
||||
|
||||
__Chrome:__ contributed by [Rpsl](https://github.com/rpsl). You can install it from [Google Chrome Webstore](https://chrome.google.com/webstore/detail/metube-downloader/fbmkmdnlhacefjljljlbhkodfmfkijdh) or use developer mode and install [from sources](https://github.com/Rpsl/metube-browser-extension).
|
||||
__Bookmarklets__ send the currently open page to MeTube with one click. Add the origins of the sites where you use them to `CORS_ALLOWED_ORIGINS`, e.g. `https://www.youtube.com,https://www.vimeo.com`. The code (Chrome and Firefox variants, contributed by [kushfest](https://github.com/kushfest) and [shoonya75](https://github.com/shoonya75)) is in the [Bookmarklets wiki page](https://github.com/alexta69/metube/wiki/Bookmarklets).
|
||||
|
||||
__Firefox:__ contributed by [nanocortex](https://github.com/nanocortex). You can install it from [Firefox Addons](https://addons.mozilla.org/en-US/firefox/addon/metube-downloader) or get sources from [here](https://github.com/nanocortex/metube-firefox-addon).
|
||||
__iOS Shortcut:__ [rithask](https://github.com/rithask) created an [iOS shortcut](https://www.icloud.com/shortcuts/66627a9f334c467baabdb2769763a1a6) for sending URLs to MeTube from Safari's share menu; it prompts for your instance address on first use.
|
||||
|
||||
## 📱 iOS Shortcut
|
||||
__Raycast:__ [dotvhs](https://github.com/dotvhs) has created an [extension for Raycast](https://www.raycast.com/dot/metube) for adding videos to MeTube directly from Raycast.
|
||||
|
||||
[rithask](https://github.com/rithask) created an iOS shortcut to send URLs to MeTube from Safari. Enter the MeTube instance address when prompted which will be saved for later use. You can run the shortcut from Safari’s share menu. The shortcut can be downloaded from [this iCloud link](https://www.icloud.com/shortcuts/66627a9f334c467baabdb2769763a1a6).
|
||||
## 🎵 Pairing with a music tagger
|
||||
|
||||
## 🔖 Bookmarklet
|
||||
MeTube deliberately stops once the file is written — tagging and library organization belong to dedicated tools. Point one at your audio download folder (`AUDIO_DOWNLOAD_DIR`):
|
||||
|
||||
[kushfest](https://github.com/kushfest) has created a Chrome bookmarklet for sending the currently open webpage to MeTube. Please note that if you're on an HTTPS page, your MeTube instance must be configured with `HTTPS` as `true` in the environment, or be behind an HTTPS reverse proxy (see below) for the bookmarklet to work.
|
||||
|
||||
Since bookmarklets run in the context of the current page (e.g. youtube.com), the requests they make to MeTube are cross-origin. You must add the origins of sites where you use the bookmarklet to the __CORS_ALLOWED_ORIGINS__ environment variable, otherwise the browser will block the requests. For example, to use the bookmarklet on YouTube and Vimeo: `CORS_ALLOWED_ORIGINS=https://www.youtube.com,https://www.vimeo.com`.
|
||||
|
||||
GitHub doesn't allow embedding JavaScript as a link, so the bookmarklet has to be created manually by copying the following code to a new bookmark you create on your bookmarks bar. Change the hostname in the URL below to point to your MeTube instance.
|
||||
|
||||
```javascript
|
||||
javascript:!function(){xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.withCredentials=true;xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function(){if(xhr.status==200){alert("Sent to metube!")}else{alert("Send to metube failed. Check the javascript console for clues.")}}}();
|
||||
```
|
||||
|
||||
[shoonya75](https://github.com/shoonya75) has contributed a Firefox version:
|
||||
|
||||
```javascript
|
||||
javascript:(function(){xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function(){if(xhr.status==200){alert("Sent to metube!")}else{alert("Send to metube failed. Check the javascript console for clues.")}}})();
|
||||
```
|
||||
|
||||
The above bookmarklets use `alert()` for notifications. This variant shows a toast instead (Chrome — for Firefox, replace the `!function(){...}()` wrapper with `(function(){...})()`):
|
||||
|
||||
```javascript
|
||||
javascript:!function(){function notify(msg) {var sc = document.scrollingElement.scrollTop; var text = document.createElement('span');text.innerHTML=msg;var ts = text.style;ts.all = 'revert';ts.color = '#000';ts.fontFamily = 'Verdana, sans-serif';ts.fontSize = '15px';ts.backgroundColor = 'white';ts.padding = '15px';ts.border = '1px solid gainsboro';ts.boxShadow = '3px 3px 10px';ts.zIndex = '100';document.body.appendChild(text);ts.position = 'absolute'; ts.top = 50 + sc + 'px'; ts.left = (window.innerWidth / 2)-(text.offsetWidth / 2) + 'px'; setTimeout(function () { text.style.visibility = "hidden"; }, 1500);}xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function() { if(xhr.status==200){notify("Sent to metube!")}else {notify("Send to metube failed. Check the javascript console for clues.")}}}();
|
||||
```
|
||||
|
||||
## ⚡ Raycast extension
|
||||
|
||||
[dotvhs](https://github.com/dotvhs) has created an [extension for Raycast](https://www.raycast.com/dot/metube) for adding videos to MeTube directly from Raycast.
|
||||
* [beets](https://beets.io) — `beet import` matches tracks against MusicBrainz, fixes tags, and files them into an Artist/Album library; headless and scriptable.
|
||||
* [MusicBrainz Picard](https://picard.musicbrainz.org) — GUI tagger with acoustic fingerprinting.
|
||||
* [Lidarr](https://lidarr.audio) — full music library manager; add the folder as an import path.
|
||||
|
||||
## 🔒 HTTPS support, and running behind a reverse proxy
|
||||
|
||||
@@ -293,11 +273,7 @@ services:
|
||||
- KEYFILE=/ssl/key.pem
|
||||
```
|
||||
|
||||
MeTube can also run behind a reverse proxy for HTTPS termination or authentication. When serving under a subdirectory, set `URL_PREFIX` accordingly.
|
||||
|
||||
The [linuxserver/swag](https://docs.linuxserver.io/general/swag) image includes ready-made snippets for MeTube in [subfolder](https://github.com/linuxserver/reverse-proxy-confs/blob/master/metube.subfolder.conf.sample) and [subdomain](https://github.com/linuxserver/reverse-proxy-confs/blob/master/metube.subdomain.conf.sample) modes, plus Authelia for authentication.
|
||||
|
||||
### 🌐 NGINX
|
||||
MeTube can also run behind a reverse proxy for HTTPS termination or authentication. When serving under a subdirectory, set `URL_PREFIX` accordingly. MeTube uses WebSocket for real-time updates, so the proxy must pass the `Upgrade`/`Connection` headers, as in this NGINX example:
|
||||
|
||||
```nginx
|
||||
location /metube/ {
|
||||
@@ -309,41 +285,7 @@ location /metube/ {
|
||||
}
|
||||
```
|
||||
|
||||
Note: the extra `proxy_set_header` directives are there to make WebSocket work.
|
||||
|
||||
### 🌐 Apache
|
||||
|
||||
Contributed by [PIE-yt](https://github.com/PIE-yt). Source [here](https://gist.github.com/PIE-yt/29e7116588379032427f5bd446b2cac4).
|
||||
|
||||
```apache
|
||||
# For putting in your Apache sites site.conf
|
||||
# Serves MeTube under a /metube/ subdir (http://yourdomain.com/metube/)
|
||||
<Location /metube/>
|
||||
ProxyPass http://localhost:8081/ retry=0 timeout=30
|
||||
ProxyPassReverse http://localhost:8081/
|
||||
</Location>
|
||||
|
||||
<Location /metube/socket.io>
|
||||
RewriteEngine On
|
||||
RewriteCond %{QUERY_STRING} transport=websocket [NC]
|
||||
RewriteRule /(.*) ws://localhost:8081/socket.io/$1 [P,L]
|
||||
ProxyPass http://localhost:8081/socket.io retry=0 timeout=30
|
||||
ProxyPassReverse http://localhost:8081/socket.io
|
||||
</Location>
|
||||
```
|
||||
|
||||
### 🌐 Caddy
|
||||
|
||||
The following example Caddyfile gets a reverse proxy going behind [caddy](https://caddyserver.com).
|
||||
|
||||
```caddyfile
|
||||
example.com {
|
||||
route /metube/* {
|
||||
uri strip_prefix metube
|
||||
reverse_proxy metube:8081
|
||||
}
|
||||
}
|
||||
```
|
||||
Apache, Caddy, and [linuxserver/swag](https://docs.linuxserver.io/general/swag) (with Authelia) examples are in the [Reverse proxy configurations wiki page](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations).
|
||||
|
||||
## 🔄 Updating yt-dlp
|
||||
|
||||
@@ -358,9 +300,11 @@ docker exec -ti metube sh
|
||||
cd /downloads
|
||||
```
|
||||
|
||||
Common issues and their fixes are collected in the [Troubleshooting FAQ](https://github.com/alexta69/metube/wiki/Troubleshooting-FAQ) on the wiki.
|
||||
|
||||
## 💡 Submitting feature requests
|
||||
|
||||
MeTube development relies on community contributions. If you need additional features, please submit a PR. Create an issue first to discuss the implementation — some PRs may not be accepted to reduce bloat. Feature requests without an accompanying PR are unlikely to be fulfilled.
|
||||
MeTube development relies on community contributions. If you need additional features, please submit a PR. Create an issue first to discuss the implementation before writing code — MeTube's scope is deliberately narrow: it downloads well and stops once the file is written. Features that improve the download itself are welcome; post-download file management (tag editing, metadata lookups, library organization) is out of scope regardless of implementation quality — see [AGENTS.md](AGENTS.md) for the full policy. Feature requests without an accompanying PR are unlikely to be fulfilled.
|
||||
|
||||
## 🛠️ Building and running locally
|
||||
|
||||
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
# Security Policy
|
||||
|
||||
## Reporting a vulnerability
|
||||
|
||||
Please report vulnerabilities privately via
|
||||
[GitHub private vulnerability reporting](https://github.com/alexta69/metube/security/advisories/new)
|
||||
(Security tab → "Report a vulnerability"). Do **not** open a public issue for
|
||||
security problems.
|
||||
|
||||
You can expect an initial response within a few days. Please include a
|
||||
reproduction and the MeTube release version (visible in the UI footer).
|
||||
|
||||
## Supported versions
|
||||
|
||||
MeTube is continuously released; only the **latest release** is supported.
|
||||
Update to the current Docker image before reporting.
|
||||
|
||||
## Scope notes
|
||||
|
||||
MeTube ships **without authentication by design** — it is intended to run on a
|
||||
trusted network or behind an authenticating reverse proxy (see the
|
||||
[wiki](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations)).
|
||||
Reports that reduce to "the UI is reachable without a login" are expected
|
||||
behavior, not vulnerabilities.
|
||||
@@ -19,6 +19,7 @@ import yt_dlp.networking.impersonate
|
||||
import bg_tasks
|
||||
from dl_formats import merge_ytdl_option_layers
|
||||
from state_store import AtomicJsonStore, read_legacy_shelf
|
||||
from url_guard import validate_url
|
||||
|
||||
log = logging.getLogger("subscriptions")
|
||||
|
||||
@@ -113,6 +114,9 @@ def extract_flat_playlist(
|
||||
nested_url = _entry_video_url(ent)
|
||||
if not nested_url:
|
||||
continue
|
||||
# nested_url comes from remote playlist content; guard it too.
|
||||
if validate_url(nested_url) is not None:
|
||||
continue
|
||||
nested_info, nested_entries = extract_flat_playlist(
|
||||
config,
|
||||
nested_url,
|
||||
@@ -542,6 +546,12 @@ class SubscriptionManager:
|
||||
url = self._normalize_url(url)
|
||||
if not url:
|
||||
return {"status": "error", "msg": "Missing URL"}
|
||||
# SSRF guard: block non-http(s) schemes and internal/metadata hosts
|
||||
# before yt-dlp fetches the feed. May do a DNS lookup, so run off-loop.
|
||||
url_error = await asyncio.get_running_loop().run_in_executor(None, validate_url, url)
|
||||
if url_error is not None:
|
||||
log.warning('Rejected subscription URL "%s": %s', url, url_error)
|
||||
return {"status": "error", "msg": url_error}
|
||||
try:
|
||||
title_regex_stored = validate_title_regex(title_regex)
|
||||
except re.error as exc:
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
"""Tests for the SSRF URL guard (``url_guard.validate_url``)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
from url_guard import validate_url
|
||||
|
||||
|
||||
def _addrinfo(*addrs, family=socket.AF_INET):
|
||||
return [(family, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", (addr, 0)) for addr in addrs]
|
||||
|
||||
|
||||
class NonUrlInputTests(unittest.TestCase):
|
||||
"""Bare IDs and yt-dlp search/extractor prefixes must pass untouched."""
|
||||
|
||||
def test_bare_video_id_allowed(self):
|
||||
self.assertIsNone(validate_url("dQw4w9WgXcQ"))
|
||||
|
||||
def test_ytsearch_prefix_allowed(self):
|
||||
self.assertIsNone(validate_url("ytsearch:some song"))
|
||||
|
||||
def test_empty_string_allowed(self):
|
||||
self.assertIsNone(validate_url(""))
|
||||
|
||||
def test_non_string_rejected(self):
|
||||
self.assertIsNotNone(validate_url(None))
|
||||
|
||||
|
||||
class SchemeTests(unittest.TestCase):
|
||||
def test_file_scheme_blocked(self):
|
||||
self.assertIsNotNone(validate_url("file:///etc/passwd"))
|
||||
|
||||
def test_ftp_scheme_blocked(self):
|
||||
self.assertIsNotNone(validate_url("ftp://example.com/x"))
|
||||
|
||||
def test_data_scheme_blocked(self):
|
||||
self.assertIsNotNone(validate_url("data://text/plain;base64,AAAA"))
|
||||
|
||||
|
||||
class HostnameBlocklistTests(unittest.TestCase):
|
||||
def test_localhost_blocked_without_lookup(self):
|
||||
with mock.patch("url_guard.socket.getaddrinfo") as gai:
|
||||
self.assertIsNotNone(validate_url("http://localhost:8080/x"))
|
||||
gai.assert_not_called()
|
||||
|
||||
def test_localhost_subdomain_blocked(self):
|
||||
self.assertIsNotNone(validate_url("http://foo.localhost/x"))
|
||||
|
||||
def test_gcp_metadata_name_blocked(self):
|
||||
self.assertIsNotNone(validate_url("http://metadata.google.internal/x"))
|
||||
|
||||
|
||||
class AddressResolutionTests(unittest.TestCase):
|
||||
def _validate_with_addrs(self, url, *addrs, family=socket.AF_INET):
|
||||
with mock.patch("url_guard.socket.getaddrinfo", return_value=_addrinfo(*addrs, family=family)):
|
||||
return validate_url(url)
|
||||
|
||||
def test_public_https_allowed(self):
|
||||
self.assertIsNone(self._validate_with_addrs("https://youtube.com/watch?v=x", "142.250.1.1"))
|
||||
|
||||
def test_public_http_allowed(self):
|
||||
self.assertIsNone(self._validate_with_addrs("http://example.com/x", "93.184.216.34"))
|
||||
|
||||
def test_link_local_metadata_blocked(self):
|
||||
self.assertIsNotNone(self._validate_with_addrs("http://metadata/x", "169.254.169.254"))
|
||||
|
||||
def test_loopback_ipv4_blocked(self):
|
||||
self.assertIsNotNone(self._validate_with_addrs("http://127.0.0.1/x", "127.0.0.1"))
|
||||
|
||||
def test_private_rfc1918_blocked(self):
|
||||
self.assertIsNotNone(self._validate_with_addrs("http://intranet/x", "10.0.0.5"))
|
||||
|
||||
def test_decimal_ip_form_blocked(self):
|
||||
# 2852039166 == 169.254.169.254; the OS resolver normalizes it.
|
||||
self.assertIsNotNone(self._validate_with_addrs("http://2852039166/x", "169.254.169.254"))
|
||||
|
||||
def test_ipv6_loopback_blocked(self):
|
||||
self.assertIsNotNone(
|
||||
self._validate_with_addrs("http://[::1]/x", "::1", family=socket.AF_INET6)
|
||||
)
|
||||
|
||||
def test_ipv4_mapped_ipv6_metadata_blocked(self):
|
||||
self.assertIsNotNone(
|
||||
self._validate_with_addrs(
|
||||
"http://evil/x", "::ffff:169.254.169.254", family=socket.AF_INET6
|
||||
)
|
||||
)
|
||||
|
||||
def test_mixed_public_and_private_blocked(self):
|
||||
# If any resolved address is internal, reject the whole URL.
|
||||
self.assertIsNotNone(self._validate_with_addrs("http://mixed/x", "142.250.1.1", "127.0.0.1"))
|
||||
|
||||
def test_resolution_failure_defers_to_ytdlp(self):
|
||||
with mock.patch("url_guard.socket.getaddrinfo", side_effect=socket.gaierror):
|
||||
self.assertIsNone(validate_url("http://does-not-resolve.example/x"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -15,6 +15,8 @@ from unittest.mock import MagicMock, patch
|
||||
fake_yt_dlp = types.ModuleType("yt_dlp")
|
||||
fake_networking = types.ModuleType("yt_dlp.networking")
|
||||
fake_impersonate = types.ModuleType("yt_dlp.networking.impersonate")
|
||||
fake_postprocessor = types.ModuleType("yt_dlp.postprocessor")
|
||||
fake_postprocessor_common = types.ModuleType("yt_dlp.postprocessor.common")
|
||||
fake_utils = types.ModuleType("yt_dlp.utils")
|
||||
|
||||
|
||||
@@ -24,25 +26,36 @@ class _ImpersonateTarget:
|
||||
return value
|
||||
|
||||
|
||||
class _PostProcessor:
|
||||
def __init__(self, downloader=None):
|
||||
self._downloader = downloader
|
||||
|
||||
|
||||
fake_impersonate.ImpersonateTarget = _ImpersonateTarget
|
||||
fake_networking.impersonate = fake_impersonate
|
||||
fake_postprocessor_common.PostProcessor = _PostProcessor
|
||||
# The inner ``key`` group mirrors the real ``STR_FORMAT_RE_TMPL`` so that
|
||||
# ``_OUTTMPL_FIELD_RE`` (compiled at import time) has the named group that
|
||||
# ``_resolve_outtmpl_fields`` reads via ``match.group('key')``.
|
||||
fake_utils.STR_FORMAT_RE_TMPL = r"(?P<prefix>)%\((?P<has_key>(?P<key>{}))\)(?P<format>[-0-9.]*{})"
|
||||
fake_utils.STR_FORMAT_TYPES = "diouxXeEfFgGcrsa"
|
||||
fake_yt_dlp.networking = fake_networking
|
||||
fake_yt_dlp.postprocessor = fake_postprocessor
|
||||
fake_yt_dlp.utils = fake_utils
|
||||
sys.modules.setdefault("yt_dlp", fake_yt_dlp)
|
||||
sys.modules.setdefault("yt_dlp.networking", fake_networking)
|
||||
sys.modules.setdefault("yt_dlp.networking.impersonate", fake_impersonate)
|
||||
sys.modules.setdefault("yt_dlp.postprocessor", fake_postprocessor)
|
||||
sys.modules.setdefault("yt_dlp.postprocessor.common", fake_postprocessor_common)
|
||||
sys.modules.setdefault("yt_dlp.utils", fake_utils)
|
||||
|
||||
import ytdl
|
||||
from ytdl import (
|
||||
Download,
|
||||
DownloadInfo,
|
||||
_compact_persisted_entry,
|
||||
_convert_srt_to_txt_file,
|
||||
_AlbumArtistPostProcessor,
|
||||
_output_dir_escapes,
|
||||
_resolve_outtmpl_fields,
|
||||
_sanitize_entry_for_pickle,
|
||||
@@ -54,6 +67,132 @@ from ytdl import (
|
||||
_has_real_ytdlp = hasattr(sys.modules.get("yt_dlp"), "YoutubeDL")
|
||||
|
||||
|
||||
class AlbumArtistPostProcessorTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.postprocessor = _AlbumArtistPostProcessor()
|
||||
|
||||
def test_fills_album_artist_from_artist(self):
|
||||
info = {'album': 'CrasH Talk', 'artist': 'ScHoolboy Q'}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'ScHoolboy Q')
|
||||
|
||||
def test_uses_main_artist_for_featured_track(self):
|
||||
info = {
|
||||
'album': 'CrasH Talk',
|
||||
'artists': ['ScHoolboy Q · Travis Scott'],
|
||||
}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'ScHoolboy Q')
|
||||
|
||||
def test_uses_topic_channel_artist_for_joint_album(self):
|
||||
info = {
|
||||
'album': 'Watch the Throne',
|
||||
'artists': ['JAY-Z', 'Kanye West'],
|
||||
'channel': 'JAY-Z & Kanye West - Topic',
|
||||
}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'JAY-Z & Kanye West')
|
||||
|
||||
def test_uses_topic_uploader_and_strips_suffix_for_compilation(self):
|
||||
info = {
|
||||
'album': 'Compilation',
|
||||
'artist': 'Track Artist',
|
||||
'channel': 'Regular Channel',
|
||||
'uploader': 'Various Artists - Topic',
|
||||
}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'Various Artists')
|
||||
|
||||
def test_regular_channel_falls_back_to_main_artist(self):
|
||||
info = {
|
||||
'album': 'Album',
|
||||
'artist': 'Track Artist',
|
||||
'channel': 'Label Channel',
|
||||
}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'Track Artist')
|
||||
|
||||
def test_preserves_explicit_various_artists(self):
|
||||
info = {
|
||||
'album': 'Revenge of the Dreamers III',
|
||||
'artist': 'J. Cole',
|
||||
'album_artist': 'Various Artists',
|
||||
}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'Various Artists')
|
||||
|
||||
def test_preserves_existing_album_artists_list(self):
|
||||
info = {
|
||||
'album': 'Album',
|
||||
'artist': 'Track Artist',
|
||||
'album_artists': ['Album Artist'],
|
||||
}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artists'], ['Album Artist'])
|
||||
self.assertNotIn('album_artist', result)
|
||||
|
||||
def test_uses_first_artist_when_artist_list_has_multiple_entries(self):
|
||||
info = {'album': 'Album', 'artists': ['Main Artist', 'Featured Artist']}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertEqual(result['album_artist'], 'Main Artist')
|
||||
|
||||
def test_does_not_fill_without_album(self):
|
||||
info = {'artist': 'Standalone Artist'}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertNotIn('album_artist', result)
|
||||
self.assertNotIn('album_artists', result)
|
||||
|
||||
def test_does_not_fill_without_artist(self):
|
||||
info = {'album': 'Instrumental Album'}
|
||||
|
||||
_, result = self.postprocessor.run(info)
|
||||
|
||||
self.assertNotIn('album_artist', result)
|
||||
self.assertNotIn('album_artists', result)
|
||||
|
||||
|
||||
class AlbumArtistRegistrationTests(unittest.TestCase):
|
||||
def test_audio_download_registers_pre_process_postprocessor(self):
|
||||
download = _make_test_download()
|
||||
download.info.download_type = 'audio'
|
||||
fake_ydl = MagicMock()
|
||||
|
||||
with patch('ytdl.yt_dlp.YoutubeDL', return_value=fake_ydl):
|
||||
result = download._make_youtube_dl({'quiet': True})
|
||||
|
||||
self.assertIs(result, fake_ydl)
|
||||
postprocessor, = fake_ydl.add_post_processor.call_args.args
|
||||
self.assertIsInstance(postprocessor, _AlbumArtistPostProcessor)
|
||||
self.assertEqual(fake_ydl.add_post_processor.call_args.kwargs, {'when': 'pre_process'})
|
||||
|
||||
def test_video_download_does_not_register_postprocessor(self):
|
||||
download = _make_test_download()
|
||||
fake_ydl = MagicMock()
|
||||
|
||||
with patch('ytdl.yt_dlp.YoutubeDL', return_value=fake_ydl):
|
||||
download._make_youtube_dl({'quiet': True})
|
||||
|
||||
fake_ydl.add_post_processor.assert_not_called()
|
||||
|
||||
|
||||
class SanitizePathComponentTests(unittest.TestCase):
|
||||
def test_replaces_windows_invalid_chars(self):
|
||||
self.assertEqual(_sanitize_path_component('a:b*c?d"e<f>g|h'), "a_b_c_d_e_f_g_h")
|
||||
@@ -267,11 +406,16 @@ class ProgressThrottleTests(unittest.TestCase):
|
||||
|
||||
|
||||
class CancelProcessGroupTests(unittest.TestCase):
|
||||
def test_cancel_kills_group_when_child_is_group_leader(self):
|
||||
# cancel() now sends SIGINT first (so yt-dlp/ffmpeg can finalize the
|
||||
# partial file) and schedules a SIGKILL escalation via the event loop
|
||||
# after ytdl._CANCEL_GRACE_SECONDS, instead of SIGKILLing immediately.
|
||||
|
||||
def test_cancel_sends_sigint_to_group_and_schedules_sigkill_escalation(self):
|
||||
# Child successfully ran os.setpgrp(): its pgid equals its own pid.
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321)
|
||||
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
|
||||
dl.loop = MagicMock()
|
||||
|
||||
with patch.object(Download, "running", return_value=True), \
|
||||
patch("ytdl.os.getpgid", return_value=4321) as mock_getpgid, \
|
||||
@@ -279,38 +423,109 @@ class CancelProcessGroupTests(unittest.TestCase):
|
||||
dl.cancel()
|
||||
|
||||
mock_getpgid.assert_called_once_with(4321)
|
||||
mock_killpg.assert_called_once_with(4321, signal.SIGKILL)
|
||||
mock_killpg.assert_called_once_with(4321, signal.SIGINT)
|
||||
dl.loop.call_later.assert_called_once_with(ytdl._CANCEL_GRACE_SECONDS, dl._kill_if_alive)
|
||||
self.assertTrue(dl.canceled)
|
||||
|
||||
def test_cancel_does_not_killpg_parent_group_kills_child_only(self):
|
||||
def test_cancel_does_not_killpg_parent_group_signals_child_only(self):
|
||||
# Child has NOT become its own group leader yet (pgid != pid, e.g. it is
|
||||
# still in the server's process group). killpg must NOT be called — that
|
||||
# would SIGKILL the whole server — and we fall back to proc.kill().
|
||||
# would signal the whole server — and we fall back to os.kill(pid, SIGINT).
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321, kill=MagicMock())
|
||||
dl.proc = types.SimpleNamespace(pid=4321)
|
||||
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
|
||||
dl.loop = MagicMock()
|
||||
|
||||
with patch.object(Download, "running", return_value=True), \
|
||||
patch("ytdl.os.getpgid", return_value=999), \
|
||||
patch("ytdl.os.killpg") as mock_killpg:
|
||||
patch("ytdl.os.killpg") as mock_killpg, \
|
||||
patch("ytdl.os.kill") as mock_kill:
|
||||
dl.cancel()
|
||||
|
||||
mock_killpg.assert_not_called()
|
||||
dl.proc.kill.assert_called_once()
|
||||
mock_kill.assert_called_once_with(4321, signal.SIGINT)
|
||||
dl.loop.call_later.assert_called_once_with(ytdl._CANCEL_GRACE_SECONDS, dl._kill_if_alive)
|
||||
self.assertTrue(dl.canceled)
|
||||
|
||||
def test_cancel_falls_back_to_proc_kill_when_getpgid_unavailable(self):
|
||||
def test_cancel_falls_back_to_pid_signal_when_getpgid_unavailable(self):
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321)
|
||||
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
|
||||
dl.loop = MagicMock()
|
||||
|
||||
with patch.object(Download, "running", return_value=True), \
|
||||
patch("ytdl.os.getpgid", side_effect=OSError("no such process")), \
|
||||
patch("ytdl.os.kill") as mock_kill:
|
||||
dl.cancel()
|
||||
|
||||
mock_kill.assert_called_once_with(4321, signal.SIGINT)
|
||||
dl.loop.call_later.assert_called_once_with(ytdl._CANCEL_GRACE_SECONDS, dl._kill_if_alive)
|
||||
self.assertTrue(dl.canceled)
|
||||
|
||||
def test_cancel_kills_immediately_when_signal_delivery_fails(self):
|
||||
# Neither killpg nor os.kill succeed (process already gone): cancel()
|
||||
# must fall through to an immediate SIGKILL attempt instead of
|
||||
# scheduling a pointless escalation.
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321, kill=MagicMock())
|
||||
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
|
||||
dl.loop = MagicMock()
|
||||
|
||||
with patch.object(Download, "running", return_value=True), \
|
||||
patch("ytdl.os.getpgid", side_effect=OSError("no such process")):
|
||||
patch("ytdl.os.getpgid", side_effect=OSError("no such process")), \
|
||||
patch("ytdl.os.kill", side_effect=ProcessLookupError()):
|
||||
dl.cancel()
|
||||
|
||||
dl.loop.call_later.assert_not_called()
|
||||
dl.proc.kill.assert_called_once()
|
||||
self.assertTrue(dl.canceled)
|
||||
|
||||
def test_cancel_schedules_escalation_even_without_running_loop(self):
|
||||
# dl.loop is None (e.g. cancel() called before start()'s run_in_executor
|
||||
# set it up): _kill_if_alive() must run synchronously instead of being
|
||||
# scheduled, since there's no loop to schedule it on.
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321)
|
||||
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
|
||||
self.assertIsNone(dl.loop)
|
||||
|
||||
with patch.object(Download, "running", side_effect=[True, True]), \
|
||||
patch("ytdl.os.getpgid", return_value=4321), \
|
||||
patch("ytdl.os.killpg") as mock_killpg:
|
||||
dl.cancel()
|
||||
|
||||
# First SIGINT, then _kill_if_alive() ran inline and sent SIGKILL.
|
||||
mock_killpg.assert_has_calls([
|
||||
unittest.mock.call(4321, signal.SIGINT),
|
||||
unittest.mock.call(4321, signal.SIGKILL),
|
||||
])
|
||||
self.assertTrue(dl.canceled)
|
||||
|
||||
|
||||
class KillIfAliveTests(unittest.TestCase):
|
||||
def test_kill_if_alive_sigkills_running_process(self):
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321)
|
||||
|
||||
with patch.object(Download, "running", return_value=True), \
|
||||
patch("ytdl.os.getpgid", return_value=4321), \
|
||||
patch("ytdl.os.killpg") as mock_killpg:
|
||||
dl._kill_if_alive()
|
||||
|
||||
mock_killpg.assert_called_once_with(4321, signal.SIGKILL)
|
||||
|
||||
def test_kill_if_alive_noop_when_process_already_exited(self):
|
||||
dl = _make_test_download()
|
||||
dl.proc = types.SimpleNamespace(pid=4321)
|
||||
|
||||
with patch.object(Download, "running", return_value=False), \
|
||||
patch("ytdl.os.killpg") as mock_killpg, \
|
||||
patch("ytdl.os.kill") as mock_kill:
|
||||
dl._kill_if_alive()
|
||||
|
||||
mock_killpg.assert_not_called()
|
||||
mock_kill.assert_not_called()
|
||||
|
||||
|
||||
class ConvertSrtToTxtTests(unittest.TestCase):
|
||||
def test_basic_conversion(self):
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
"""Lightweight SSRF guard for user-submitted URLs.
|
||||
|
||||
MeTube hands user-submitted URLs to yt-dlp, whose generic extractor will fetch
|
||||
any ``http(s)`` URL. Without a guard, an attacker can make the server fetch
|
||||
internal endpoints (cloud metadata services, loopback, RFC1918 hosts, etc.) and
|
||||
have the response saved to the download directory and served back.
|
||||
|
||||
This module provides a single cheap validator applied at every URL ingress. It
|
||||
intentionally does NOT attempt DNS-rebinding pinning, redirect-chain
|
||||
re-validation, or validation of every media URL yt-dlp derives from remote
|
||||
metadata — network isolation (e.g. Docker) remains the backstop for those.
|
||||
"""
|
||||
|
||||
import ipaddress
|
||||
import logging
|
||||
import socket
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
log = logging.getLogger('url_guard')
|
||||
|
||||
_ALLOWED_SCHEMES = ('http', 'https')
|
||||
|
||||
# Hostnames that must be blocked without needing a lookup. ``localhost`` and any
|
||||
# subdomain of it are conventionally loopback, and the GCP metadata name is a
|
||||
# well-known SSRF target that may resolve via a resolver we don't control.
|
||||
_BLOCKED_HOSTNAMES = ('localhost', 'metadata.google.internal')
|
||||
|
||||
|
||||
def _hostname_is_blocked(hostname: str) -> bool:
|
||||
host = hostname.rstrip('.').lower()
|
||||
for blocked in _BLOCKED_HOSTNAMES:
|
||||
if host == blocked or host.endswith('.' + blocked):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _address_is_global(addr: str) -> bool:
|
||||
try:
|
||||
ip = ipaddress.ip_address(addr)
|
||||
except ValueError:
|
||||
return False
|
||||
# Unwrap IPv4-mapped/compatible IPv6 (e.g. ::ffff:169.254.169.254) so the
|
||||
# embedded IPv4 address is judged on its own merits.
|
||||
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
|
||||
ip = ip.ipv4_mapped
|
||||
return ip.is_global
|
||||
|
||||
|
||||
def validate_url(url: str) -> str | None:
|
||||
"""Return an error message if the URL is disallowed, else ``None``.
|
||||
|
||||
Inputs without a ``://`` scheme separator (bare video IDs, ``ytsearch:``
|
||||
and other yt-dlp search/extractor prefixes) are allowed unchanged so that
|
||||
non-URL entries keep working.
|
||||
"""
|
||||
if not isinstance(url, str):
|
||||
return 'Invalid URL'
|
||||
|
||||
candidate = url.strip()
|
||||
if '://' not in candidate:
|
||||
# Not an absolute URL: bare video IDs, ytsearch: prefixes, etc.
|
||||
return None
|
||||
|
||||
parts = urlsplit(candidate)
|
||||
scheme = parts.scheme.lower()
|
||||
if scheme not in _ALLOWED_SCHEMES:
|
||||
return f'URL scheme "{parts.scheme}" is not allowed (only http and https)'
|
||||
|
||||
hostname = parts.hostname
|
||||
if not hostname:
|
||||
return 'URL is missing a host'
|
||||
|
||||
if _hostname_is_blocked(hostname):
|
||||
return f'Refusing to fetch internal host "{hostname}"'
|
||||
|
||||
try:
|
||||
addrinfo = socket.getaddrinfo(hostname, parts.port, proto=socket.IPPROTO_TCP)
|
||||
except socket.gaierror:
|
||||
# Let yt-dlp surface a normal resolution error rather than masking it.
|
||||
return None
|
||||
except (UnicodeError, ValueError):
|
||||
return f'Invalid host "{hostname}"'
|
||||
|
||||
for family, _type, _proto, _canonname, sockaddr in addrinfo:
|
||||
addr = sockaddr[0]
|
||||
if not _address_is_global(addr):
|
||||
return f'Refusing to fetch internal address "{addr}" for host "{hostname}"'
|
||||
|
||||
return None
|
||||
+107
-21
@@ -19,12 +19,14 @@ import types
|
||||
from typing import Any, Optional
|
||||
|
||||
import yt_dlp.networking.impersonate
|
||||
from yt_dlp.postprocessor.common import PostProcessor
|
||||
from yt_dlp.utils import STR_FORMAT_RE_TMPL, STR_FORMAT_TYPES
|
||||
import bg_tasks
|
||||
from dl_formats import get_format, get_opts, AUDIO_FORMATS, merge_ytdl_option_layers
|
||||
from datetime import datetime
|
||||
from state_store import AtomicJsonStore, from_json_compatible, read_legacy_shelf, to_json_compatible
|
||||
from subscriptions import _entry_id
|
||||
from url_guard import validate_url
|
||||
|
||||
log = logging.getLogger('ytdl')
|
||||
|
||||
@@ -45,6 +47,10 @@ _MP_CTX = (
|
||||
else multiprocessing.get_context()
|
||||
)
|
||||
|
||||
# Grace period between SIGINT (lets yt-dlp/ffmpeg finalize the partial file,
|
||||
# e.g. when cancelling a livestream) and SIGKILL escalation.
|
||||
_CANCEL_GRACE_SECONDS = 15
|
||||
|
||||
_LIVE_CHECK_INTERVAL = 60
|
||||
_LIVE_MAX_CHECK_INTERVAL = 3600
|
||||
# Consecutive probe failures (network blips, rate limits, transient extractor
|
||||
@@ -52,6 +58,52 @@ _LIVE_MAX_CHECK_INTERVAL = 3600
|
||||
_LIVE_PROBE_MAX_FAILURES = 5
|
||||
|
||||
|
||||
class _AlbumArtistPostProcessor(PostProcessor):
|
||||
"""Fill missing album-artist metadata from yt-dlp's album-level signals."""
|
||||
|
||||
_TOPIC_SUFFIX = ' - Topic'
|
||||
|
||||
@staticmethod
|
||||
def _has_value(value: Any) -> bool:
|
||||
if isinstance(value, str):
|
||||
return bool(value.strip())
|
||||
if isinstance(value, (list, tuple)):
|
||||
return any(_AlbumArtistPostProcessor._has_value(item) for item in value)
|
||||
return value is not None
|
||||
|
||||
@staticmethod
|
||||
def _main_artist(info) -> Optional[str]:
|
||||
artists = info.get('artists')
|
||||
candidates = artists if isinstance(artists, list) else [info.get('artist')]
|
||||
for candidate in candidates:
|
||||
if not isinstance(candidate, str) or not candidate.strip():
|
||||
continue
|
||||
# YouTube Music uses a spaced middle dot between credited artists.
|
||||
# The first credit is the primary artist for normal albums.
|
||||
return candidate.split(' · ', 1)[0].strip()
|
||||
return None
|
||||
|
||||
@classmethod
|
||||
def _topic_artist(cls, info) -> Optional[str]:
|
||||
for field in ('channel', 'uploader'):
|
||||
value = info.get(field)
|
||||
if not isinstance(value, str) or not value.endswith(cls._TOPIC_SUFFIX):
|
||||
continue
|
||||
if artist := value[:-len(cls._TOPIC_SUFFIX)].strip():
|
||||
return artist
|
||||
return None
|
||||
|
||||
def run(self, info):
|
||||
if not self._has_value(info.get('album')):
|
||||
return [], info
|
||||
if self._has_value(info.get('album_artist')) or self._has_value(info.get('album_artists')):
|
||||
return [], info
|
||||
|
||||
if artist := self._topic_artist(info) or self._main_artist(info):
|
||||
info['album_artist'] = artist
|
||||
return [], info
|
||||
|
||||
|
||||
def _is_within_directory(real_base: str, real_target: str) -> bool:
|
||||
"""True if ``real_target`` is inside (or equal to) ``real_base``.
|
||||
|
||||
@@ -545,6 +597,12 @@ class Download:
|
||||
|
||||
return put_status
|
||||
|
||||
def _make_youtube_dl(self, params):
|
||||
ydl = yt_dlp.YoutubeDL(params=params)
|
||||
if getattr(self.info, 'download_type', '') == 'audio':
|
||||
ydl.add_post_processor(_AlbumArtistPostProcessor(ydl), when='pre_process')
|
||||
return ydl
|
||||
|
||||
def _download(self):
|
||||
# Run in our own process group so cancel() can SIGKILL the whole
|
||||
# group (yt-dlp + any ffmpeg children it spawned for merge/postproc),
|
||||
@@ -622,7 +680,7 @@ class Download:
|
||||
[(start, end)],
|
||||
)
|
||||
|
||||
ret = yt_dlp.YoutubeDL(params=ytdl_params).download([self.info.url])
|
||||
ret = self._make_youtube_dl(ytdl_params).download([self.info.url])
|
||||
self.status_queue.put({'status': 'finished' if ret == 0 else 'error'})
|
||||
log.info(f"Finished download for: {self.info.title}")
|
||||
except yt_dlp.utils.YoutubeDLError as exc:
|
||||
@@ -650,29 +708,50 @@ class Download:
|
||||
self.status_queue.put(None)
|
||||
await self.status_task
|
||||
|
||||
def _signal_group(self, sig):
|
||||
"""Send *sig* to the download's process group, falling back to the
|
||||
process itself. Returns True if a signal was delivered.
|
||||
|
||||
Only signal the whole group when the child actually became its own
|
||||
group leader via os.setpgrp() in _download() — that sets its pgid
|
||||
equal to its own pid. If it hasn't run setpgrp() yet, or setpgrp()
|
||||
failed, its pgid is still the SERVER's group and killpg would signal
|
||||
the entire MeTube process (PID 1 in Docker). Fall back to signalling
|
||||
just the child process by pid in that case.
|
||||
"""
|
||||
try:
|
||||
pgid = os.getpgid(self.proc.pid)
|
||||
if pgid == self.proc.pid:
|
||||
os.killpg(pgid, sig)
|
||||
return True
|
||||
except (OSError, AttributeError):
|
||||
pass
|
||||
try:
|
||||
if sig == signal.SIGINT:
|
||||
os.kill(self.proc.pid, sig)
|
||||
else:
|
||||
self.proc.kill()
|
||||
return True
|
||||
except Exception as e:
|
||||
log.error(f"Error signalling process for {self.info.title}: {e}")
|
||||
return False
|
||||
|
||||
def _kill_if_alive(self):
|
||||
if self.running():
|
||||
log.info(f"Escalating cancel to SIGKILL for: {self.info.title}")
|
||||
self._signal_group(signal.SIGKILL)
|
||||
|
||||
def cancel(self):
|
||||
log.info(f"Cancelling download: {self.info.title}")
|
||||
if self.running():
|
||||
killed_group = False
|
||||
try:
|
||||
pgid = os.getpgid(self.proc.pid)
|
||||
# Only kill the whole group (yt-dlp + any ffmpeg children) when
|
||||
# the child actually became its own group leader via
|
||||
# os.setpgrp() in _download() — that sets its pgid equal to its
|
||||
# own pid. If it hasn't run setpgrp() yet, or setpgrp() failed,
|
||||
# its pgid is still the SERVER's group and killpg would SIGKILL
|
||||
# the entire MeTube process (PID 1 in Docker). Fall back to
|
||||
# killing just the child process by pid in that case.
|
||||
if pgid == self.proc.pid:
|
||||
os.killpg(pgid, signal.SIGKILL)
|
||||
killed_group = True
|
||||
except (OSError, AttributeError):
|
||||
pass
|
||||
if not killed_group:
|
||||
try:
|
||||
self.proc.kill()
|
||||
except Exception as e:
|
||||
log.error(f"Error killing process for {self.info.title}: {e}")
|
||||
# SIGINT first so yt-dlp/ffmpeg can finalize the partial file
|
||||
# (livestream recordings stay playable); SIGKILL after a grace
|
||||
# period if the process ignores it.
|
||||
interrupted = self._signal_group(signal.SIGINT)
|
||||
if interrupted and self.loop is not None:
|
||||
self.loop.call_later(_CANCEL_GRACE_SECONDS, self._kill_if_alive)
|
||||
else:
|
||||
self._kill_if_alive()
|
||||
self.canceled = True
|
||||
if self.status_queue is not None:
|
||||
self.status_queue.put(None)
|
||||
@@ -1444,6 +1523,13 @@ class DownloadQueue:
|
||||
return {'status': 'ok'}
|
||||
else:
|
||||
already.add(url)
|
||||
# SSRF guard: reject non-http(s) schemes and hosts resolving to
|
||||
# internal/loopback/link-local/metadata addresses before yt-dlp fetches
|
||||
# anything. run_in_executor because validate_url may perform a DNS lookup.
|
||||
url_error = await asyncio.get_running_loop().run_in_executor(None, validate_url, url)
|
||||
if url_error is not None:
|
||||
log.warning('Rejected URL "%s": %s', url, url_error)
|
||||
return {'status': 'error', 'msg': url_error}
|
||||
try:
|
||||
entry = await asyncio.get_running_loop().run_in_executor(
|
||||
None,
|
||||
|
||||
+8
-8
@@ -33,10 +33,10 @@
|
||||
"@angular/platform-browser-dynamic": "^22.0.6",
|
||||
"@angular/service-worker": "^22.0.6",
|
||||
"@fortawesome/angular-fontawesome": "~4.0.0",
|
||||
"@fortawesome/fontawesome-svg-core": "^7.3.0",
|
||||
"@fortawesome/free-brands-svg-icons": "^7.3.0",
|
||||
"@fortawesome/free-regular-svg-icons": "^7.3.0",
|
||||
"@fortawesome/free-solid-svg-icons": "^7.3.0",
|
||||
"@fortawesome/fontawesome-svg-core": "^7.3.1",
|
||||
"@fortawesome/free-brands-svg-icons": "^7.3.1",
|
||||
"@fortawesome/free-regular-svg-icons": "^7.3.1",
|
||||
"@fortawesome/free-solid-svg-icons": "^7.3.1",
|
||||
"@ng-bootstrap/ng-bootstrap": "^21.0.0",
|
||||
"@ng-select/ng-select": "^23.2.0",
|
||||
"@popperjs/core": "^2.11.8",
|
||||
@@ -49,13 +49,13 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@angular-eslint/builder": "22.0.0",
|
||||
"@angular/build": "^22.0.5",
|
||||
"@angular/cli": "^22.0.5",
|
||||
"@angular/build": "^22.0.7",
|
||||
"@angular/cli": "^22.0.7",
|
||||
"@angular/compiler-cli": "^22.0.6",
|
||||
"@angular/localize": "^22.0.6",
|
||||
"@eslint/js": "^9.39.4",
|
||||
"@eslint/js": "^9.39.5",
|
||||
"angular-eslint": "22.0.0",
|
||||
"eslint": "^9.39.4",
|
||||
"eslint": "^9.39.5",
|
||||
"jsdom": "^27.4.0",
|
||||
"typescript": "~6.0.3",
|
||||
"typescript-eslint": "8.62.0",
|
||||
|
||||
Generated
+258
-285
File diff suppressed because it is too large
Load Diff
@@ -106,14 +106,14 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "anyio"
|
||||
version = "4.14.1"
|
||||
version = "4.14.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "idna" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/3b/72/5562aabb8dd7181e8e860622a38bea08d17842b99ecd4c91f84ac95251b0/anyio-4.14.1.tar.gz", hash = "sha256:8d648a3544c1a700e3ff78615cd679e4c5c3f149904287e73687b2596963629e", size = 254831, upload-time = "2026-06-24T20:56:06.017Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/61/cc/a381afa6efea9f496eff839d4a6a1aed3bfafc7b3ab4b0d1b243a12573dd/anyio-4.14.2.tar.gz", hash = "sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f", size = 260176, upload-time = "2026-07-12T20:29:07.082Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b0/7b/90df4a0a816d98d6ea26f559d87836d494a2cf1fcf063be67df50a7bcc30/anyio-4.14.1-py3-none-any.whl", hash = "sha256:4e5533c5b8ff0a24f5d7a176cbe6877129cd183893f66b537f8f227d10527d72", size = 124875, upload-time = "2026-06-24T20:56:04.413Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -366,15 +366,15 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "deno"
|
||||
version = "2.9.2"
|
||||
version = "2.9.3"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/dc/3c/bf3a4e2c00f7eefd4ed2fd033a958cab3e92dcc539d3ab80351850cae99c/deno-2.9.2.tar.gz", hash = "sha256:3b77d7689c15fb2c47d5a3927dc8cf70dbd408bc92e20c70da079c6d78980e34", size = 8164, upload-time = "2026-07-08T14:38:54.267Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/98/ab/638749d76881f74d100a079414745f0a0fd20bba38e27ab4a3a6495d8264/deno-2.9.3.tar.gz", hash = "sha256:73268cbac7f7c4ff1983e49420a93f3f3f2f2e4e4372436f0942d04d02c9e943", size = 8166, upload-time = "2026-07-15T15:37:50.565Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/27/fe/4a02a256f19b3f31ac17ec37f6467ecf523601ed753e6c9177865dd2bbb2/deno-2.9.2-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:fb4bb5362d02c193f8086668789076f366416980b79277c06ea6cff73866b668", size = 42341183, upload-time = "2026-07-08T14:38:36.384Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5a/3e/235530ac9b9c206ee8c098e2b215f711a71d915d8d7d2be77599c8c1db4b/deno-2.9.2-py3-none-macosx_11_0_arm64.whl", hash = "sha256:f6f74fd8698a8a9a31af14eed5730f1964ab50eaa0129079d11d91fa92496f60", size = 37984508, upload-time = "2026-07-08T14:38:40.006Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/7b/27/06a166fca538c06dfae4c45dbb92c2ebf3083341141966a3d649272ea734/deno-2.9.2-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:8b2fd9318d452b7e44c45968916670a0c5452d72796e1af9abfaa7a389e4aa6b", size = 42089854, upload-time = "2026-07-08T14:38:43.841Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/09/b4/38338550a5ca0a7bcc39991e3d48e46ce3699413460dff32f383a611d822/deno-2.9.2-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:bf0471db7238e90810bca77a8d2c28646317b5a3fd9e5cdfdf73b251fde11b83", size = 43916921, upload-time = "2026-07-08T14:38:48.057Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a9/3d/c490c27e1f720baf014bb2c18b29d09b11c21d69fadfd69639e5904cdc95/deno-2.9.2-py3-none-win_amd64.whl", hash = "sha256:8e815d66b3e1314d028b2f0a05c75a331dd7025f4c825e12dd1f057207cee1f6", size = 41621982, upload-time = "2026-07-08T14:38:51.928Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/07/fa/b4e1e2b3b894ed992c78a6796a43dc775edff895239d83c0e9a10457f4f7/deno-2.9.3-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:1deef9fe97ab28d5d4fea07693075e9ccc62f4ea97a653047a770beb01ecd307", size = 42354026, upload-time = "2026-07-15T15:37:34.977Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/3d/cc/4090ef7370005b740efe57ec6ff75a3471393abe9816f1459e73dbbf0429/deno-2.9.3-py3-none-macosx_11_0_arm64.whl", hash = "sha256:78635cd9f0802ce7125aa3ba6feb5775460f55d432611374f16b454ffb8a1308", size = 37997909, upload-time = "2026-07-15T15:37:38.291Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/14/07/70b7c965bbca5f3d158859acda0e90c041f1fa98b8c9ac8eff3fafd8b96c/deno-2.9.3-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:7c1c669827a480ef9e2710dd762560811da35dca04a9e90b23c0518587dadb8a", size = 42100873, upload-time = "2026-07-15T15:37:41.539Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/7b/f5/0a07cc19a27476011719dd017a98950c7ccc33058c2ffc07019ba3ad9a5e/deno-2.9.3-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:3e724d5c8df13a90f6b50b16cc8df1acfe64c61a7736a774e11d61f3bca3a3b1", size = 43928967, upload-time = "2026-07-15T15:37:45.175Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e1/9a/030847bd4ea6cefbb471cae2b63c9ab29a3ab6c1809484447752bbcbcd52/deno-2.9.3-py3-none-win_amd64.whl", hash = "sha256:e2ebe2b5c1a7ee9daeaf8caf14ebc33fda96b4c2399da259e761fe2f82d22fa4", size = 41630201, upload-time = "2026-07-15T15:37:48.344Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
Reference in New Issue
Block a user