Compare commits

...

13 Commits

Author SHA1 Message Date
Alex Shnitman a23d1689e3 docs: add SECURITY.md (private vulnerability reporting)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 09:24:04 +03:00
Alex Shnitman 4b05022b91 feat: graceful cancel — SIGINT with SIGKILL escalation so partial files are finalized (closes #438)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 09:23:48 +03:00
Alex Shnitman 0dc9b0b3d6 ci: enforce the 25k Docker Hub limit on README.md
Runs only when README.md changes (which the build workflow ignores),
so the limit is checked exactly when it can be exceeded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:39:31 +03:00
Alex Shnitman 96f5ffe34a docs: restructure README around the wiki as companion documentation
- Move bookmarklet code and Apache/Caddy/swag reverse-proxy configs to
  the wiki; link them
- Group browser extensions, bookmarklets, iOS Shortcut, and Raycast
  under one 'Sending links to MeTube' section with shared CORS/HTTPS
  prerequisites stated once
- Move Runtime & Permissions (PUID/PGID/UMASK) to the top of the
  env-var reference; slim the CORS_ALLOWED_ORIGINS entry
- Link the new Subscriptions guide and Troubleshooting FAQ wiki pages
- State the project scope line in 'Submitting feature requests'
- Docker Compose naming, multi-arch note, trailing whitespace

24,830 -> 21,571 chars against the 25k Docker Hub limit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:39:20 +03:00
Alex Shnitman 1eafecd1cc docs: add test gotchas, option checklist, and security invariants to AGENTS.md
- Test-running traps: repo-root cwd requirement, frontend-build ordering,
  the yt_dlp stub quirk in test_ytdl_utils.py
- Note that master is continuously released
- Checklist for adding a per-download option (the split_by_chapters
  pattern, including the commonly missed pieces)
- Security invariants: SSRF guard for URLs, path helpers for anything
  derived from untrusted metadata
- Conventions: OnPush/markForCheck, compact persisted state, yt-dlp
  postprocessor list ordering

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:14:33 +03:00
Alex Shnitman 8fe81dea18 docs: encode project scope boundary in AGENTS.md
Document the line between in-scope work (improving the download-time
write, surfacing yt-dlp built-ins) and out-of-scope work (post-download
tag editing, external metadata lookups, library organization), so
agent-assisted contributions can check their plans against it before
writing code. Follows the decisions on #1025, #1026/#1027, #1028, #1031.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:08:33 +03:00
Alex f054d84108 Merge pull request #1029 from tjelite1986/readme-library-pairing
docs: document pairing MeTube with a music tagger
2026-07-17 15:00:51 +03:00
tjelite1986 fa02717e12 docs: document pairing MeTube with a music tagger
Short section pointing beets / Picard / Lidarr at AUDIO_DOWNLOAD_DIR,
as suggested in #1027.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 22:39:24 +02:00
Alex Shnitman 9ca78be199 Merge PR #1025: fill missing album-artist metadata for audio downloads
Adds _AlbumArtistPostProcessor, a yt-dlp pre_process postprocessor that
fills album_artist from the '<artist> - Topic' channel/uploader signal,
falling back to the first credited artist, when album metadata exists
but no album artist is set.

Closes #1025.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 23:03:17 +03:00
Alex Shnitman 8071611a84 upgrade dependencies 2026-07-16 23:01:23 +03:00
Your GitHub Name 220f991fae fix: prefer topic channel for album artist 2026-07-16 12:34:03 -07:00
Alex Shnitman 6d0528783c fix: block SSRF via user-submitted URLs
User-submitted URLs were passed straight to yt-dlp's generic extractor,
letting the server fetch internal endpoints (cloud metadata, loopback,
RFC1918 hosts). Add a url_guard.validate_url check at every URL ingress
(add, subscribe, and nested playlist recursion) that rejects non-http(s)
schemes and hosts resolving to non-global addresses, while leaving bare
video IDs and search prefixes untouched.
2026-07-16 21:31:51 +03:00
Your GitHub Name c104e30451 feat: add AlbumArtistPostProcessor to fill missing album-artist metadata 2026-07-15 15:40:36 -07:00
12 changed files with 983 additions and 422 deletions
+21
View File
@@ -0,0 +1,21 @@
name: readme-size
on:
push:
paths:
- 'README.md'
pull_request:
paths:
- 'README.md'
jobs:
check-size:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v7
- name: Check README stays under Docker Hub's 25k character limit
run: |
size=$(wc -c < README.md)
echo "README.md is ${size} bytes (limit 25000)"
test "$size" -lt 25000
+97
View File
@@ -1,5 +1,53 @@
# Agent Guidelines
## Project scope — read this before planning a feature
MeTube's contract is: give it a URL, it runs yt-dlp well, and correct files appear.
The maintainer holds a deliberate line on what belongs inside that contract, and PRs
on the wrong side of it are declined **regardless of code quality**. Check your plan
against this line before writing any code.
**In scope — improving the write itself:**
- Features that make the file yt-dlp writes at download time come out more correct,
using only data the extractor already provides (e.g. filling a missing album-artist
tag from the extractor's own metadata).
- Surfacing functionality yt-dlp itself owns and maintains as first-class UI options
(e.g. a SponsorBlock toggle that just passes yt-dlp postprocessor params).
- Download queue, subscriptions, output templates, and UI improvements to the
download workflow.
**Out of scope — managing files after they exist:**
- Tag editors, metadata dialogs, or any workflow that rewrites files after the
download has finished. This holds even for slimmed-down versions.
- Lookups against external metadata services (iTunes, Deezer, MusicBrainz, etc.).
More broadly: any new dependency on a third-party API, or new network egress from
self-hosted instances, beyond what yt-dlp itself performs.
- Library organization: moving/renaming existing files into Artist/Album layouts,
watch-folder processing, and similar media-manager features. Dedicated tools
(beets, MusicBrainz Picard, Lidarr) do this properly; the README points users
to them.
**Corollaries that shape borderline PRs:**
- Site-specific intelligence (parsing playlist-ID prefixes, URL path conventions,
and other platform internals) is extractor work and belongs upstream in yt-dlp,
not re-implemented here — it silently breaks when the platform changes and
MeTube would own the breakage.
- Prefer enriching yt-dlp's info dict and letting its existing pipeline
(FFmpegMetadata etc.) do the writing, over adding custom per-format tag-writing
code to MeTube.
- Supplemental processing must never fail a download that otherwise succeeded:
warn and continue, don't raise.
- Keep feature scope minimal on first submission. A hardcoded sensible default
beats a configuration surface; follow-ups can add options when users actually
ask. PRs that bundle several "reasonable next steps" invite rejection of the
whole.
If a feature idea fails this test, the accepted alternative is usually a README
section documenting how to pair MeTube with the right dedicated tool.
## README.md size constraint
The README.md is synced to Docker Hub, which has a **25,000 character limit**.
@@ -30,6 +78,24 @@ uv run pytest app/tests/
All of these run in CI (`.github/workflows/main.yml`) on every push to master and must pass.
Gotchas:
- Backend tests must run **from the repo root**: `main.py` resolves the static-assets
path relative to the cwd, and several test modules import `main`. Running from
`app/` makes five test modules fail to import.
- The frontend must be **built before** running backend tests (same reason — the
assets at `ui/dist/metube/browser` must exist). The command order above is
load-bearing.
- `app/tests/test_ytdl_utils.py` stubs `yt_dlp` at import time. Run standalone,
two tests fail with `AttributeError: <module 'yt_dlp'> does not have the
attribute 'YoutubeDL'`; under the full suite the real module is imported first
and they pass. This is a known quirk, not a bug to fix in the code under test.
Every non-markdown push to master builds multi-arch Docker images and cuts a dated
release the same day. **Master is continuously released** — a PR must be
release-ready exactly as merged; there is no stabilization window for follow-up
fixes.
## Code style
Follow `.editorconfig`:
@@ -56,5 +122,36 @@ ui/src/app/ — Angular standalone components (no NgModules)
- Backend configuration lives in the `Config` class in `app/main.py` with env-var defaults in `_DEFAULTS`. New env vars go there.
- Real-time communication uses Socket.IO events, not REST polling.
- Frontend uses standalone Angular components with `inject()` for DI, RxJS Subjects for state, and `takeUntilDestroyed()` for cleanup.
- Frontend components use OnPush change detection: subscribe callbacks must call `cdr.markForCheck()`.
- State is persisted as JSON files via `AtomicJsonStore` in `app/state_store.py`.
- Persisted state stays compact: the completed queue deliberately drops bulky entry data (see `_compact_persisted_entry` in `app/ytdl.py`). Don't expand what gets persisted without discussion.
- Custom yt-dlp postprocessors added to `ytdl_params['postprocessors']` run in **list order** within a stage. When combining postprocessors, mirror the ordering the yt-dlp CLI would produce (e.g. sponsor-segment removal before chapter splitting).
- No pre-commit hooks — linting and tests are enforced in CI only.
## Checklist: adding a per-download option
New options on the download form (the `split_by_chapters` pattern) need **all** of
these pieces — the last three are the ones commonly missed:
1. `parse_download_options` in `app/main.py`.
2. A field on `DownloadInfo` in `app/ytdl.py`.
3. A `hasattr` backfill in `DownloadInfo.__setstate__` for old persisted records.
4. The safe-deserialization field list in `app/ytdl.py`.
5. UI form control + cookie persistence in `ui/src/app/app.ts` / `app.html`, and
the payload in `downloads.service.ts` (plus its spec).
6. The redownload path in `app.ts`, so retries carry the option.
7. If the option makes sense for unattended downloads: threading through
`app/subscriptions.py` (`SubscriptionInfo` field, serializer, add/update
routes, the enqueue call) — or a note in the PR that it's deliberately
direct-downloads-only.
## Security invariants
User input and extractor-provided metadata (titles, playlist names, URLs) are
untrusted. Use the existing guards instead of hand-rolling:
- User-submitted URLs go through the SSRF guard (see `test_url_guard.py` for the
expected behavior).
- Anything that becomes a filesystem path goes through `_is_within_directory` and
`_sanitize_path_component` in `app/ytdl.py` — including values that arrive via
yt-dlp metadata, which sites can influence.
+33 -89
View File
@@ -3,12 +3,12 @@
![Build Status](https://github.com/alexta69/metube/actions/workflows/main.yml/badge.svg)
![Docker Pulls](https://img.shields.io/docker/pulls/alexta69/metube.svg)
MeTube is a self-hosted web UI for `yt-dlp`, for downloading media from YouTube and [dozens of other sites](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md).
MeTube is a self-hosted web UI for `yt-dlp`, for downloading media from YouTube and [dozens of other sites](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md). Docker images are multi-arch (amd64/arm64).
Key capabilities:
* Download videos, audio, captions, and thumbnails from a browser UI.
* Download playlists and channels, with configurable output and download options.
* Subscribe to channels and playlists, periodically check for new items, and queue new uploads automatically.
* [Subscribe](https://github.com/alexta69/metube/wiki/Subscriptions) to channels and playlists, periodically check for new items, and queue new uploads automatically.
![screenshot1](https://github.com/alexta69/metube/raw/master/screenshot.gif)
@@ -18,7 +18,7 @@ Key capabilities:
docker run -d -p 8081:8081 -v /path/to/downloads:/downloads ghcr.io/alexta69/metube
```
## 🐳 Run using docker-compose
## 🐳 Run using Docker Compose
```yaml
services:
@@ -34,11 +34,20 @@ services:
## ⚙️ Configuration via environment variables
Certain values can be set via environment variables, using the `-e` parameter on the docker command line, or the `environment:` section in docker-compose.
Certain values can be set via environment variables, using the `-e` parameter on the docker command line, or the `environment:` section in Docker Compose.
### 🏠 Runtime & Permissions
* __PUID__: User under which MeTube will run. Defaults to `1000` (legacy `UID` also supported).
* __PGID__: Group under which MeTube will run. Defaults to `1000` (legacy `GID` also supported).
* __UMASK__: Umask value used by MeTube. Defaults to `022`.
* __DEFAULT_THEME__: Default theme to use for the UI, can be set to `light`, `dark`, or `auto`. Defaults to `auto`.
* __LOGLEVEL__: Log level, can be set to `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, or `NONE`. Defaults to `INFO`.
* __ENABLE_ACCESSLOG__: Whether to enable access log. Defaults to `false`.
### ⬇️ Download Behavior
* __MAX_CONCURRENT_DOWNLOADS__: Maximum number of simultaneous downloads allowed. For example, if set to `5`, then at most five downloads will run concurrently, and any additional downloads will wait until one of the active downloads completes. Defaults to `3`.
* __MAX_CONCURRENT_DOWNLOADS__: Maximum number of simultaneous downloads allowed. For example, if set to `5`, then at most five downloads will run concurrently, and any additional downloads will wait until one of the active downloads completes. Defaults to `3`.
* __DELETE_FILE_ON_TRASHCAN__: if `true`, downloaded files are deleted on the server, when they are trashed from the "Completed" section of the UI. Defaults to `false`.
* __DEFAULT_OPTION_PLAYLIST_ITEM_LIMIT__: Maximum number of playlist items that can be downloaded. Defaults to `0` (no limit).
* __SUBSCRIPTION_DEFAULT_CHECK_INTERVAL__: Default minutes between automatic checks for each subscription. Defaults to `60`.
@@ -83,18 +92,9 @@ Certain values can be set via environment variables, using the `-e` parameter on
* __HTTPS__: Use `https` instead of `http` (__CERTFILE__ and __KEYFILE__ required). Defaults to `false`.
* __CERTFILE__: HTTPS certificate file path.
* __KEYFILE__: HTTPS key file path.
* __CORS_ALLOWED_ORIGINS__: Comma-separated list of origins permitted to make cross-origin requests to the MeTube API. When unset or empty, all cross-origin requests are denied. Set to `*` to allow all origins. This must be configured for [browser extensions](#-browser-extensions), [bookmarklets](#-bookmarklet), and any other browser-based tools that contact MeTube from a different origin. For browser extensions use `*` (see below); for bookmarklets you can list specific sites, e.g. `https://www.youtube.com,https://www.vimeo.com`.
* __CORS_ALLOWED_ORIGINS__: Comma-separated list of origins permitted to make cross-origin requests to the MeTube API; `*` allows all. When unset or empty, all cross-origin requests are denied. Required for browser extensions and bookmarklets — see [Sending links to MeTube](#-sending-links-to-metube).
* __ROBOTS_TXT__: A path to a `robots.txt` file mounted in the container.
### 🏠 Basic Setup
* __PUID__: User under which MeTube will run. Defaults to `1000` (legacy `UID` also supported).
* __PGID__: Group under which MeTube will run. Defaults to `1000` (legacy `GID` also supported).
* __UMASK__: Umask value used by MeTube. Defaults to `022`.
* __DEFAULT_THEME__: Default theme to use for the UI, can be set to `light`, `dark`, or `auto`. Defaults to `auto`.
* __LOGLEVEL__: Log level, can be set to `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, or `NONE`. Defaults to `INFO`.
* __ENABLE_ACCESSLOG__: Whether to enable access log. Defaults to `false`.
## 🎛️ Configuring yt-dlp options
MeTube lets you customize how [yt-dlp](https://github.com/yt-dlp/yt-dlp) behaves at three levels, from broadest to most specific:
@@ -229,47 +229,27 @@ In case you need to use your browser's cookies with MeTube, for example to downl
* After upload, the cookie indicator should show as active.
* Use **Delete Cookies** in the same section to remove uploaded cookies.
## 🔌 Browser extensions
## 🔗 Sending links to MeTube
Browser extensions allow right-clicking videos and sending them directly to MeTube. If you're on an HTTPS page, your MeTube instance must be behind an HTTPS reverse proxy (see below) for extensions to work.
Several integrations let you send URLs to MeTube from wherever you are, instead of pasting them into the UI. The browser-based ones make cross-origin requests, so they require `CORS_ALLOWED_ORIGINS` to be set; and if you're on an HTTPS page, your MeTube instance must be served over HTTPS too (with `HTTPS=true` or behind an HTTPS reverse proxy see below).
Since browser extensions make requests from their own origin (`chrome-extension://...` or `moz-extension://...`), you must set `CORS_ALLOWED_ORIGINS=*` for them to work.
__Browser extensions__ allow right-clicking videos and sending them directly to MeTube. Since extensions request from their own origin, set `CORS_ALLOWED_ORIGINS=*`.
* __Chrome:__ contributed by [Rpsl](https://github.com/rpsl) — install from the [Chrome Webstore](https://chrome.google.com/webstore/detail/metube-downloader/fbmkmdnlhacefjljljlbhkodfmfkijdh) or [from sources](https://github.com/Rpsl/metube-browser-extension).
* __Firefox:__ contributed by [nanocortex](https://github.com/nanocortex) — install from [Firefox Addons](https://addons.mozilla.org/en-US/firefox/addon/metube-downloader) or get sources [here](https://github.com/nanocortex/metube-firefox-addon).
__Chrome:__ contributed by [Rpsl](https://github.com/rpsl). You can install it from [Google Chrome Webstore](https://chrome.google.com/webstore/detail/metube-downloader/fbmkmdnlhacefjljljlbhkodfmfkijdh) or use developer mode and install [from sources](https://github.com/Rpsl/metube-browser-extension).
__Bookmarklets__ send the currently open page to MeTube with one click. Add the origins of the sites where you use them to `CORS_ALLOWED_ORIGINS`, e.g. `https://www.youtube.com,https://www.vimeo.com`. The code (Chrome and Firefox variants, contributed by [kushfest](https://github.com/kushfest) and [shoonya75](https://github.com/shoonya75)) is in the [Bookmarklets wiki page](https://github.com/alexta69/metube/wiki/Bookmarklets).
__Firefox:__ contributed by [nanocortex](https://github.com/nanocortex). You can install it from [Firefox Addons](https://addons.mozilla.org/en-US/firefox/addon/metube-downloader) or get sources from [here](https://github.com/nanocortex/metube-firefox-addon).
__iOS Shortcut:__ [rithask](https://github.com/rithask) created an [iOS shortcut](https://www.icloud.com/shortcuts/66627a9f334c467baabdb2769763a1a6) for sending URLs to MeTube from Safari's share menu; it prompts for your instance address on first use.
## 📱 iOS Shortcut
__Raycast:__ [dotvhs](https://github.com/dotvhs) has created an [extension for Raycast](https://www.raycast.com/dot/metube) for adding videos to MeTube directly from Raycast.
[rithask](https://github.com/rithask) created an iOS shortcut to send URLs to MeTube from Safari. Enter the MeTube instance address when prompted which will be saved for later use. You can run the shortcut from Safaris share menu. The shortcut can be downloaded from [this iCloud link](https://www.icloud.com/shortcuts/66627a9f334c467baabdb2769763a1a6).
## 🎵 Pairing with a music tagger
## 🔖 Bookmarklet
MeTube deliberately stops once the file is written — tagging and library organization belong to dedicated tools. Point one at your audio download folder (`AUDIO_DOWNLOAD_DIR`):
[kushfest](https://github.com/kushfest) has created a Chrome bookmarklet for sending the currently open webpage to MeTube. Please note that if you're on an HTTPS page, your MeTube instance must be configured with `HTTPS` as `true` in the environment, or be behind an HTTPS reverse proxy (see below) for the bookmarklet to work.
Since bookmarklets run in the context of the current page (e.g. youtube.com), the requests they make to MeTube are cross-origin. You must add the origins of sites where you use the bookmarklet to the __CORS_ALLOWED_ORIGINS__ environment variable, otherwise the browser will block the requests. For example, to use the bookmarklet on YouTube and Vimeo: `CORS_ALLOWED_ORIGINS=https://www.youtube.com,https://www.vimeo.com`.
GitHub doesn't allow embedding JavaScript as a link, so the bookmarklet has to be created manually by copying the following code to a new bookmark you create on your bookmarks bar. Change the hostname in the URL below to point to your MeTube instance.
```javascript
javascript:!function(){xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.withCredentials=true;xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function(){if(xhr.status==200){alert("Sent to metube!")}else{alert("Send to metube failed. Check the javascript console for clues.")}}}();
```
[shoonya75](https://github.com/shoonya75) has contributed a Firefox version:
```javascript
javascript:(function(){xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function(){if(xhr.status==200){alert("Sent to metube!")}else{alert("Send to metube failed. Check the javascript console for clues.")}}})();
```
The above bookmarklets use `alert()` for notifications. This variant shows a toast instead (Chrome — for Firefox, replace the `!function(){...}()` wrapper with `(function(){...})()`):
```javascript
javascript:!function(){function notify(msg) {var sc = document.scrollingElement.scrollTop; var text = document.createElement('span');text.innerHTML=msg;var ts = text.style;ts.all = 'revert';ts.color = '#000';ts.fontFamily = 'Verdana, sans-serif';ts.fontSize = '15px';ts.backgroundColor = 'white';ts.padding = '15px';ts.border = '1px solid gainsboro';ts.boxShadow = '3px 3px 10px';ts.zIndex = '100';document.body.appendChild(text);ts.position = 'absolute'; ts.top = 50 + sc + 'px'; ts.left = (window.innerWidth / 2)-(text.offsetWidth / 2) + 'px'; setTimeout(function () { text.style.visibility = "hidden"; }, 1500);}xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function() { if(xhr.status==200){notify("Sent to metube!")}else {notify("Send to metube failed. Check the javascript console for clues.")}}}();
```
## ⚡ Raycast extension
[dotvhs](https://github.com/dotvhs) has created an [extension for Raycast](https://www.raycast.com/dot/metube) for adding videos to MeTube directly from Raycast.
* [beets](https://beets.io) — `beet import` matches tracks against MusicBrainz, fixes tags, and files them into an Artist/Album library; headless and scriptable.
* [MusicBrainz Picard](https://picard.musicbrainz.org) — GUI tagger with acoustic fingerprinting.
* [Lidarr](https://lidarr.audio) — full music library manager; add the folder as an import path.
## 🔒 HTTPS support, and running behind a reverse proxy
@@ -293,11 +273,7 @@ services:
- KEYFILE=/ssl/key.pem
```
MeTube can also run behind a reverse proxy for HTTPS termination or authentication. When serving under a subdirectory, set `URL_PREFIX` accordingly.
The [linuxserver/swag](https://docs.linuxserver.io/general/swag) image includes ready-made snippets for MeTube in [subfolder](https://github.com/linuxserver/reverse-proxy-confs/blob/master/metube.subfolder.conf.sample) and [subdomain](https://github.com/linuxserver/reverse-proxy-confs/blob/master/metube.subdomain.conf.sample) modes, plus Authelia for authentication.
### 🌐 NGINX
MeTube can also run behind a reverse proxy for HTTPS termination or authentication. When serving under a subdirectory, set `URL_PREFIX` accordingly. MeTube uses WebSocket for real-time updates, so the proxy must pass the `Upgrade`/`Connection` headers, as in this NGINX example:
```nginx
location /metube/ {
@@ -309,41 +285,7 @@ location /metube/ {
}
```
Note: the extra `proxy_set_header` directives are there to make WebSocket work.
### 🌐 Apache
Contributed by [PIE-yt](https://github.com/PIE-yt). Source [here](https://gist.github.com/PIE-yt/29e7116588379032427f5bd446b2cac4).
```apache
# For putting in your Apache sites site.conf
# Serves MeTube under a /metube/ subdir (http://yourdomain.com/metube/)
<Location /metube/>
ProxyPass http://localhost:8081/ retry=0 timeout=30
ProxyPassReverse http://localhost:8081/
</Location>
<Location /metube/socket.io>
RewriteEngine On
RewriteCond %{QUERY_STRING} transport=websocket [NC]
RewriteRule /(.*) ws://localhost:8081/socket.io/$1 [P,L]
ProxyPass http://localhost:8081/socket.io retry=0 timeout=30
ProxyPassReverse http://localhost:8081/socket.io
</Location>
```
### 🌐 Caddy
The following example Caddyfile gets a reverse proxy going behind [caddy](https://caddyserver.com).
```caddyfile
example.com {
route /metube/* {
uri strip_prefix metube
reverse_proxy metube:8081
}
}
```
Apache, Caddy, and [linuxserver/swag](https://docs.linuxserver.io/general/swag) (with Authelia) examples are in the [Reverse proxy configurations wiki page](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations).
## 🔄 Updating yt-dlp
@@ -358,9 +300,11 @@ docker exec -ti metube sh
cd /downloads
```
Common issues and their fixes are collected in the [Troubleshooting FAQ](https://github.com/alexta69/metube/wiki/Troubleshooting-FAQ) on the wiki.
## 💡 Submitting feature requests
MeTube development relies on community contributions. If you need additional features, please submit a PR. Create an issue first to discuss the implementation — some PRs may not be accepted to reduce bloat. Feature requests without an accompanying PR are unlikely to be fulfilled.
MeTube development relies on community contributions. If you need additional features, please submit a PR. Create an issue first to discuss the implementation before writing code — MeTube's scope is deliberately narrow: it downloads well and stops once the file is written. Features that improve the download itself are welcome; post-download file management (tag editing, metadata lookups, library organization) is out of scope regardless of implementation quality — see [AGENTS.md](AGENTS.md) for the full policy. Feature requests without an accompanying PR are unlikely to be fulfilled.
## 🛠️ Building and running locally
+24
View File
@@ -0,0 +1,24 @@
# Security Policy
## Reporting a vulnerability
Please report vulnerabilities privately via
[GitHub private vulnerability reporting](https://github.com/alexta69/metube/security/advisories/new)
(Security tab → "Report a vulnerability"). Do **not** open a public issue for
security problems.
You can expect an initial response within a few days. Please include a
reproduction and the MeTube release version (visible in the UI footer).
## Supported versions
MeTube is continuously released; only the **latest release** is supported.
Update to the current Docker image before reporting.
## Scope notes
MeTube ships **without authentication by design** — it is intended to run on a
trusted network or behind an authenticating reverse proxy (see the
[wiki](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations)).
Reports that reduce to "the UI is reachable without a login" are expected
behavior, not vulnerabilities.
+10
View File
@@ -19,6 +19,7 @@ import yt_dlp.networking.impersonate
import bg_tasks
from dl_formats import merge_ytdl_option_layers
from state_store import AtomicJsonStore, read_legacy_shelf
from url_guard import validate_url
log = logging.getLogger("subscriptions")
@@ -113,6 +114,9 @@ def extract_flat_playlist(
nested_url = _entry_video_url(ent)
if not nested_url:
continue
# nested_url comes from remote playlist content; guard it too.
if validate_url(nested_url) is not None:
continue
nested_info, nested_entries = extract_flat_playlist(
config,
nested_url,
@@ -542,6 +546,12 @@ class SubscriptionManager:
url = self._normalize_url(url)
if not url:
return {"status": "error", "msg": "Missing URL"}
# SSRF guard: block non-http(s) schemes and internal/metadata hosts
# before yt-dlp fetches the feed. May do a DNS lookup, so run off-loop.
url_error = await asyncio.get_running_loop().run_in_executor(None, validate_url, url)
if url_error is not None:
log.warning('Rejected subscription URL "%s": %s', url, url_error)
return {"status": "error", "msg": url_error}
try:
title_regex_stored = validate_title_regex(title_regex)
except re.error as exc:
+102
View File
@@ -0,0 +1,102 @@
"""Tests for the SSRF URL guard (``url_guard.validate_url``)."""
from __future__ import annotations
import socket
import unittest
from unittest import mock
from url_guard import validate_url
def _addrinfo(*addrs, family=socket.AF_INET):
return [(family, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", (addr, 0)) for addr in addrs]
class NonUrlInputTests(unittest.TestCase):
"""Bare IDs and yt-dlp search/extractor prefixes must pass untouched."""
def test_bare_video_id_allowed(self):
self.assertIsNone(validate_url("dQw4w9WgXcQ"))
def test_ytsearch_prefix_allowed(self):
self.assertIsNone(validate_url("ytsearch:some song"))
def test_empty_string_allowed(self):
self.assertIsNone(validate_url(""))
def test_non_string_rejected(self):
self.assertIsNotNone(validate_url(None))
class SchemeTests(unittest.TestCase):
def test_file_scheme_blocked(self):
self.assertIsNotNone(validate_url("file:///etc/passwd"))
def test_ftp_scheme_blocked(self):
self.assertIsNotNone(validate_url("ftp://example.com/x"))
def test_data_scheme_blocked(self):
self.assertIsNotNone(validate_url("data://text/plain;base64,AAAA"))
class HostnameBlocklistTests(unittest.TestCase):
def test_localhost_blocked_without_lookup(self):
with mock.patch("url_guard.socket.getaddrinfo") as gai:
self.assertIsNotNone(validate_url("http://localhost:8080/x"))
gai.assert_not_called()
def test_localhost_subdomain_blocked(self):
self.assertIsNotNone(validate_url("http://foo.localhost/x"))
def test_gcp_metadata_name_blocked(self):
self.assertIsNotNone(validate_url("http://metadata.google.internal/x"))
class AddressResolutionTests(unittest.TestCase):
def _validate_with_addrs(self, url, *addrs, family=socket.AF_INET):
with mock.patch("url_guard.socket.getaddrinfo", return_value=_addrinfo(*addrs, family=family)):
return validate_url(url)
def test_public_https_allowed(self):
self.assertIsNone(self._validate_with_addrs("https://youtube.com/watch?v=x", "142.250.1.1"))
def test_public_http_allowed(self):
self.assertIsNone(self._validate_with_addrs("http://example.com/x", "93.184.216.34"))
def test_link_local_metadata_blocked(self):
self.assertIsNotNone(self._validate_with_addrs("http://metadata/x", "169.254.169.254"))
def test_loopback_ipv4_blocked(self):
self.assertIsNotNone(self._validate_with_addrs("http://127.0.0.1/x", "127.0.0.1"))
def test_private_rfc1918_blocked(self):
self.assertIsNotNone(self._validate_with_addrs("http://intranet/x", "10.0.0.5"))
def test_decimal_ip_form_blocked(self):
# 2852039166 == 169.254.169.254; the OS resolver normalizes it.
self.assertIsNotNone(self._validate_with_addrs("http://2852039166/x", "169.254.169.254"))
def test_ipv6_loopback_blocked(self):
self.assertIsNotNone(
self._validate_with_addrs("http://[::1]/x", "::1", family=socket.AF_INET6)
)
def test_ipv4_mapped_ipv6_metadata_blocked(self):
self.assertIsNotNone(
self._validate_with_addrs(
"http://evil/x", "::ffff:169.254.169.254", family=socket.AF_INET6
)
)
def test_mixed_public_and_private_blocked(self):
# If any resolved address is internal, reject the whole URL.
self.assertIsNotNone(self._validate_with_addrs("http://mixed/x", "142.250.1.1", "127.0.0.1"))
def test_resolution_failure_defers_to_ytdlp(self):
with mock.patch("url_guard.socket.getaddrinfo", side_effect=socket.gaierror):
self.assertIsNone(validate_url("http://does-not-resolve.example/x"))
if __name__ == "__main__":
unittest.main()
+224 -9
View File
@@ -15,6 +15,8 @@ from unittest.mock import MagicMock, patch
fake_yt_dlp = types.ModuleType("yt_dlp")
fake_networking = types.ModuleType("yt_dlp.networking")
fake_impersonate = types.ModuleType("yt_dlp.networking.impersonate")
fake_postprocessor = types.ModuleType("yt_dlp.postprocessor")
fake_postprocessor_common = types.ModuleType("yt_dlp.postprocessor.common")
fake_utils = types.ModuleType("yt_dlp.utils")
@@ -24,25 +26,36 @@ class _ImpersonateTarget:
return value
class _PostProcessor:
def __init__(self, downloader=None):
self._downloader = downloader
fake_impersonate.ImpersonateTarget = _ImpersonateTarget
fake_networking.impersonate = fake_impersonate
fake_postprocessor_common.PostProcessor = _PostProcessor
# The inner ``key`` group mirrors the real ``STR_FORMAT_RE_TMPL`` so that
# ``_OUTTMPL_FIELD_RE`` (compiled at import time) has the named group that
# ``_resolve_outtmpl_fields`` reads via ``match.group('key')``.
fake_utils.STR_FORMAT_RE_TMPL = r"(?P<prefix>)%\((?P<has_key>(?P<key>{}))\)(?P<format>[-0-9.]*{})"
fake_utils.STR_FORMAT_TYPES = "diouxXeEfFgGcrsa"
fake_yt_dlp.networking = fake_networking
fake_yt_dlp.postprocessor = fake_postprocessor
fake_yt_dlp.utils = fake_utils
sys.modules.setdefault("yt_dlp", fake_yt_dlp)
sys.modules.setdefault("yt_dlp.networking", fake_networking)
sys.modules.setdefault("yt_dlp.networking.impersonate", fake_impersonate)
sys.modules.setdefault("yt_dlp.postprocessor", fake_postprocessor)
sys.modules.setdefault("yt_dlp.postprocessor.common", fake_postprocessor_common)
sys.modules.setdefault("yt_dlp.utils", fake_utils)
import ytdl
from ytdl import (
Download,
DownloadInfo,
_compact_persisted_entry,
_convert_srt_to_txt_file,
_AlbumArtistPostProcessor,
_output_dir_escapes,
_resolve_outtmpl_fields,
_sanitize_entry_for_pickle,
@@ -54,6 +67,132 @@ from ytdl import (
_has_real_ytdlp = hasattr(sys.modules.get("yt_dlp"), "YoutubeDL")
class AlbumArtistPostProcessorTests(unittest.TestCase):
def setUp(self):
self.postprocessor = _AlbumArtistPostProcessor()
def test_fills_album_artist_from_artist(self):
info = {'album': 'CrasH Talk', 'artist': 'ScHoolboy Q'}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'ScHoolboy Q')
def test_uses_main_artist_for_featured_track(self):
info = {
'album': 'CrasH Talk',
'artists': ['ScHoolboy Q · Travis Scott'],
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'ScHoolboy Q')
def test_uses_topic_channel_artist_for_joint_album(self):
info = {
'album': 'Watch the Throne',
'artists': ['JAY-Z', 'Kanye West'],
'channel': 'JAY-Z & Kanye West - Topic',
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'JAY-Z & Kanye West')
def test_uses_topic_uploader_and_strips_suffix_for_compilation(self):
info = {
'album': 'Compilation',
'artist': 'Track Artist',
'channel': 'Regular Channel',
'uploader': 'Various Artists - Topic',
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'Various Artists')
def test_regular_channel_falls_back_to_main_artist(self):
info = {
'album': 'Album',
'artist': 'Track Artist',
'channel': 'Label Channel',
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'Track Artist')
def test_preserves_explicit_various_artists(self):
info = {
'album': 'Revenge of the Dreamers III',
'artist': 'J. Cole',
'album_artist': 'Various Artists',
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'Various Artists')
def test_preserves_existing_album_artists_list(self):
info = {
'album': 'Album',
'artist': 'Track Artist',
'album_artists': ['Album Artist'],
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artists'], ['Album Artist'])
self.assertNotIn('album_artist', result)
def test_uses_first_artist_when_artist_list_has_multiple_entries(self):
info = {'album': 'Album', 'artists': ['Main Artist', 'Featured Artist']}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'Main Artist')
def test_does_not_fill_without_album(self):
info = {'artist': 'Standalone Artist'}
_, result = self.postprocessor.run(info)
self.assertNotIn('album_artist', result)
self.assertNotIn('album_artists', result)
def test_does_not_fill_without_artist(self):
info = {'album': 'Instrumental Album'}
_, result = self.postprocessor.run(info)
self.assertNotIn('album_artist', result)
self.assertNotIn('album_artists', result)
class AlbumArtistRegistrationTests(unittest.TestCase):
def test_audio_download_registers_pre_process_postprocessor(self):
download = _make_test_download()
download.info.download_type = 'audio'
fake_ydl = MagicMock()
with patch('ytdl.yt_dlp.YoutubeDL', return_value=fake_ydl):
result = download._make_youtube_dl({'quiet': True})
self.assertIs(result, fake_ydl)
postprocessor, = fake_ydl.add_post_processor.call_args.args
self.assertIsInstance(postprocessor, _AlbumArtistPostProcessor)
self.assertEqual(fake_ydl.add_post_processor.call_args.kwargs, {'when': 'pre_process'})
def test_video_download_does_not_register_postprocessor(self):
download = _make_test_download()
fake_ydl = MagicMock()
with patch('ytdl.yt_dlp.YoutubeDL', return_value=fake_ydl):
download._make_youtube_dl({'quiet': True})
fake_ydl.add_post_processor.assert_not_called()
class SanitizePathComponentTests(unittest.TestCase):
def test_replaces_windows_invalid_chars(self):
self.assertEqual(_sanitize_path_component('a:b*c?d"e<f>g|h'), "a_b_c_d_e_f_g_h")
@@ -267,11 +406,16 @@ class ProgressThrottleTests(unittest.TestCase):
class CancelProcessGroupTests(unittest.TestCase):
def test_cancel_kills_group_when_child_is_group_leader(self):
# cancel() now sends SIGINT first (so yt-dlp/ffmpeg can finalize the
# partial file) and schedules a SIGKILL escalation via the event loop
# after ytdl._CANCEL_GRACE_SECONDS, instead of SIGKILLing immediately.
def test_cancel_sends_sigint_to_group_and_schedules_sigkill_escalation(self):
# Child successfully ran os.setpgrp(): its pgid equals its own pid.
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321)
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
dl.loop = MagicMock()
with patch.object(Download, "running", return_value=True), \
patch("ytdl.os.getpgid", return_value=4321) as mock_getpgid, \
@@ -279,38 +423,109 @@ class CancelProcessGroupTests(unittest.TestCase):
dl.cancel()
mock_getpgid.assert_called_once_with(4321)
mock_killpg.assert_called_once_with(4321, signal.SIGKILL)
mock_killpg.assert_called_once_with(4321, signal.SIGINT)
dl.loop.call_later.assert_called_once_with(ytdl._CANCEL_GRACE_SECONDS, dl._kill_if_alive)
self.assertTrue(dl.canceled)
def test_cancel_does_not_killpg_parent_group_kills_child_only(self):
def test_cancel_does_not_killpg_parent_group_signals_child_only(self):
# Child has NOT become its own group leader yet (pgid != pid, e.g. it is
# still in the server's process group). killpg must NOT be called — that
# would SIGKILL the whole server — and we fall back to proc.kill().
# would signal the whole server — and we fall back to os.kill(pid, SIGINT).
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321, kill=MagicMock())
dl.proc = types.SimpleNamespace(pid=4321)
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
dl.loop = MagicMock()
with patch.object(Download, "running", return_value=True), \
patch("ytdl.os.getpgid", return_value=999), \
patch("ytdl.os.killpg") as mock_killpg:
patch("ytdl.os.killpg") as mock_killpg, \
patch("ytdl.os.kill") as mock_kill:
dl.cancel()
mock_killpg.assert_not_called()
dl.proc.kill.assert_called_once()
mock_kill.assert_called_once_with(4321, signal.SIGINT)
dl.loop.call_later.assert_called_once_with(ytdl._CANCEL_GRACE_SECONDS, dl._kill_if_alive)
self.assertTrue(dl.canceled)
def test_cancel_falls_back_to_proc_kill_when_getpgid_unavailable(self):
def test_cancel_falls_back_to_pid_signal_when_getpgid_unavailable(self):
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321)
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
dl.loop = MagicMock()
with patch.object(Download, "running", return_value=True), \
patch("ytdl.os.getpgid", side_effect=OSError("no such process")), \
patch("ytdl.os.kill") as mock_kill:
dl.cancel()
mock_kill.assert_called_once_with(4321, signal.SIGINT)
dl.loop.call_later.assert_called_once_with(ytdl._CANCEL_GRACE_SECONDS, dl._kill_if_alive)
self.assertTrue(dl.canceled)
def test_cancel_kills_immediately_when_signal_delivery_fails(self):
# Neither killpg nor os.kill succeed (process already gone): cancel()
# must fall through to an immediate SIGKILL attempt instead of
# scheduling a pointless escalation.
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321, kill=MagicMock())
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
dl.loop = MagicMock()
with patch.object(Download, "running", return_value=True), \
patch("ytdl.os.getpgid", side_effect=OSError("no such process")):
patch("ytdl.os.getpgid", side_effect=OSError("no such process")), \
patch("ytdl.os.kill", side_effect=ProcessLookupError()):
dl.cancel()
dl.loop.call_later.assert_not_called()
dl.proc.kill.assert_called_once()
self.assertTrue(dl.canceled)
def test_cancel_schedules_escalation_even_without_running_loop(self):
# dl.loop is None (e.g. cancel() called before start()'s run_in_executor
# set it up): _kill_if_alive() must run synchronously instead of being
# scheduled, since there's no loop to schedule it on.
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321)
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
self.assertIsNone(dl.loop)
with patch.object(Download, "running", side_effect=[True, True]), \
patch("ytdl.os.getpgid", return_value=4321), \
patch("ytdl.os.killpg") as mock_killpg:
dl.cancel()
# First SIGINT, then _kill_if_alive() ran inline and sent SIGKILL.
mock_killpg.assert_has_calls([
unittest.mock.call(4321, signal.SIGINT),
unittest.mock.call(4321, signal.SIGKILL),
])
self.assertTrue(dl.canceled)
class KillIfAliveTests(unittest.TestCase):
def test_kill_if_alive_sigkills_running_process(self):
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321)
with patch.object(Download, "running", return_value=True), \
patch("ytdl.os.getpgid", return_value=4321), \
patch("ytdl.os.killpg") as mock_killpg:
dl._kill_if_alive()
mock_killpg.assert_called_once_with(4321, signal.SIGKILL)
def test_kill_if_alive_noop_when_process_already_exited(self):
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321)
with patch.object(Download, "running", return_value=False), \
patch("ytdl.os.killpg") as mock_killpg, \
patch("ytdl.os.kill") as mock_kill:
dl._kill_if_alive()
mock_killpg.assert_not_called()
mock_kill.assert_not_called()
class ConvertSrtToTxtTests(unittest.TestCase):
def test_basic_conversion(self):
+89
View File
@@ -0,0 +1,89 @@
"""Lightweight SSRF guard for user-submitted URLs.
MeTube hands user-submitted URLs to yt-dlp, whose generic extractor will fetch
any ``http(s)`` URL. Without a guard, an attacker can make the server fetch
internal endpoints (cloud metadata services, loopback, RFC1918 hosts, etc.) and
have the response saved to the download directory and served back.
This module provides a single cheap validator applied at every URL ingress. It
intentionally does NOT attempt DNS-rebinding pinning, redirect-chain
re-validation, or validation of every media URL yt-dlp derives from remote
metadata — network isolation (e.g. Docker) remains the backstop for those.
"""
import ipaddress
import logging
import socket
from urllib.parse import urlsplit
log = logging.getLogger('url_guard')
_ALLOWED_SCHEMES = ('http', 'https')
# Hostnames that must be blocked without needing a lookup. ``localhost`` and any
# subdomain of it are conventionally loopback, and the GCP metadata name is a
# well-known SSRF target that may resolve via a resolver we don't control.
_BLOCKED_HOSTNAMES = ('localhost', 'metadata.google.internal')
def _hostname_is_blocked(hostname: str) -> bool:
host = hostname.rstrip('.').lower()
for blocked in _BLOCKED_HOSTNAMES:
if host == blocked or host.endswith('.' + blocked):
return True
return False
def _address_is_global(addr: str) -> bool:
try:
ip = ipaddress.ip_address(addr)
except ValueError:
return False
# Unwrap IPv4-mapped/compatible IPv6 (e.g. ::ffff:169.254.169.254) so the
# embedded IPv4 address is judged on its own merits.
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
ip = ip.ipv4_mapped
return ip.is_global
def validate_url(url: str) -> str | None:
"""Return an error message if the URL is disallowed, else ``None``.
Inputs without a ``://`` scheme separator (bare video IDs, ``ytsearch:``
and other yt-dlp search/extractor prefixes) are allowed unchanged so that
non-URL entries keep working.
"""
if not isinstance(url, str):
return 'Invalid URL'
candidate = url.strip()
if '://' not in candidate:
# Not an absolute URL: bare video IDs, ytsearch: prefixes, etc.
return None
parts = urlsplit(candidate)
scheme = parts.scheme.lower()
if scheme not in _ALLOWED_SCHEMES:
return f'URL scheme "{parts.scheme}" is not allowed (only http and https)'
hostname = parts.hostname
if not hostname:
return 'URL is missing a host'
if _hostname_is_blocked(hostname):
return f'Refusing to fetch internal host "{hostname}"'
try:
addrinfo = socket.getaddrinfo(hostname, parts.port, proto=socket.IPPROTO_TCP)
except socket.gaierror:
# Let yt-dlp surface a normal resolution error rather than masking it.
return None
except (UnicodeError, ValueError):
return f'Invalid host "{hostname}"'
for family, _type, _proto, _canonname, sockaddr in addrinfo:
addr = sockaddr[0]
if not _address_is_global(addr):
return f'Refusing to fetch internal address "{addr}" for host "{hostname}"'
return None
+107 -21
View File
@@ -19,12 +19,14 @@ import types
from typing import Any, Optional
import yt_dlp.networking.impersonate
from yt_dlp.postprocessor.common import PostProcessor
from yt_dlp.utils import STR_FORMAT_RE_TMPL, STR_FORMAT_TYPES
import bg_tasks
from dl_formats import get_format, get_opts, AUDIO_FORMATS, merge_ytdl_option_layers
from datetime import datetime
from state_store import AtomicJsonStore, from_json_compatible, read_legacy_shelf, to_json_compatible
from subscriptions import _entry_id
from url_guard import validate_url
log = logging.getLogger('ytdl')
@@ -45,6 +47,10 @@ _MP_CTX = (
else multiprocessing.get_context()
)
# Grace period between SIGINT (lets yt-dlp/ffmpeg finalize the partial file,
# e.g. when cancelling a livestream) and SIGKILL escalation.
_CANCEL_GRACE_SECONDS = 15
_LIVE_CHECK_INTERVAL = 60
_LIVE_MAX_CHECK_INTERVAL = 3600
# Consecutive probe failures (network blips, rate limits, transient extractor
@@ -52,6 +58,52 @@ _LIVE_MAX_CHECK_INTERVAL = 3600
_LIVE_PROBE_MAX_FAILURES = 5
class _AlbumArtistPostProcessor(PostProcessor):
"""Fill missing album-artist metadata from yt-dlp's album-level signals."""
_TOPIC_SUFFIX = ' - Topic'
@staticmethod
def _has_value(value: Any) -> bool:
if isinstance(value, str):
return bool(value.strip())
if isinstance(value, (list, tuple)):
return any(_AlbumArtistPostProcessor._has_value(item) for item in value)
return value is not None
@staticmethod
def _main_artist(info) -> Optional[str]:
artists = info.get('artists')
candidates = artists if isinstance(artists, list) else [info.get('artist')]
for candidate in candidates:
if not isinstance(candidate, str) or not candidate.strip():
continue
# YouTube Music uses a spaced middle dot between credited artists.
# The first credit is the primary artist for normal albums.
return candidate.split(' · ', 1)[0].strip()
return None
@classmethod
def _topic_artist(cls, info) -> Optional[str]:
for field in ('channel', 'uploader'):
value = info.get(field)
if not isinstance(value, str) or not value.endswith(cls._TOPIC_SUFFIX):
continue
if artist := value[:-len(cls._TOPIC_SUFFIX)].strip():
return artist
return None
def run(self, info):
if not self._has_value(info.get('album')):
return [], info
if self._has_value(info.get('album_artist')) or self._has_value(info.get('album_artists')):
return [], info
if artist := self._topic_artist(info) or self._main_artist(info):
info['album_artist'] = artist
return [], info
def _is_within_directory(real_base: str, real_target: str) -> bool:
"""True if ``real_target`` is inside (or equal to) ``real_base``.
@@ -545,6 +597,12 @@ class Download:
return put_status
def _make_youtube_dl(self, params):
ydl = yt_dlp.YoutubeDL(params=params)
if getattr(self.info, 'download_type', '') == 'audio':
ydl.add_post_processor(_AlbumArtistPostProcessor(ydl), when='pre_process')
return ydl
def _download(self):
# Run in our own process group so cancel() can SIGKILL the whole
# group (yt-dlp + any ffmpeg children it spawned for merge/postproc),
@@ -622,7 +680,7 @@ class Download:
[(start, end)],
)
ret = yt_dlp.YoutubeDL(params=ytdl_params).download([self.info.url])
ret = self._make_youtube_dl(ytdl_params).download([self.info.url])
self.status_queue.put({'status': 'finished' if ret == 0 else 'error'})
log.info(f"Finished download for: {self.info.title}")
except yt_dlp.utils.YoutubeDLError as exc:
@@ -650,29 +708,50 @@ class Download:
self.status_queue.put(None)
await self.status_task
def _signal_group(self, sig):
"""Send *sig* to the download's process group, falling back to the
process itself. Returns True if a signal was delivered.
Only signal the whole group when the child actually became its own
group leader via os.setpgrp() in _download() — that sets its pgid
equal to its own pid. If it hasn't run setpgrp() yet, or setpgrp()
failed, its pgid is still the SERVER's group and killpg would signal
the entire MeTube process (PID 1 in Docker). Fall back to signalling
just the child process by pid in that case.
"""
try:
pgid = os.getpgid(self.proc.pid)
if pgid == self.proc.pid:
os.killpg(pgid, sig)
return True
except (OSError, AttributeError):
pass
try:
if sig == signal.SIGINT:
os.kill(self.proc.pid, sig)
else:
self.proc.kill()
return True
except Exception as e:
log.error(f"Error signalling process for {self.info.title}: {e}")
return False
def _kill_if_alive(self):
if self.running():
log.info(f"Escalating cancel to SIGKILL for: {self.info.title}")
self._signal_group(signal.SIGKILL)
def cancel(self):
log.info(f"Cancelling download: {self.info.title}")
if self.running():
killed_group = False
try:
pgid = os.getpgid(self.proc.pid)
# Only kill the whole group (yt-dlp + any ffmpeg children) when
# the child actually became its own group leader via
# os.setpgrp() in _download() — that sets its pgid equal to its
# own pid. If it hasn't run setpgrp() yet, or setpgrp() failed,
# its pgid is still the SERVER's group and killpg would SIGKILL
# the entire MeTube process (PID 1 in Docker). Fall back to
# killing just the child process by pid in that case.
if pgid == self.proc.pid:
os.killpg(pgid, signal.SIGKILL)
killed_group = True
except (OSError, AttributeError):
pass
if not killed_group:
try:
self.proc.kill()
except Exception as e:
log.error(f"Error killing process for {self.info.title}: {e}")
# SIGINT first so yt-dlp/ffmpeg can finalize the partial file
# (livestream recordings stay playable); SIGKILL after a grace
# period if the process ignores it.
interrupted = self._signal_group(signal.SIGINT)
if interrupted and self.loop is not None:
self.loop.call_later(_CANCEL_GRACE_SECONDS, self._kill_if_alive)
else:
self._kill_if_alive()
self.canceled = True
if self.status_queue is not None:
self.status_queue.put(None)
@@ -1444,6 +1523,13 @@ class DownloadQueue:
return {'status': 'ok'}
else:
already.add(url)
# SSRF guard: reject non-http(s) schemes and hosts resolving to
# internal/loopback/link-local/metadata addresses before yt-dlp fetches
# anything. run_in_executor because validate_url may perform a DNS lookup.
url_error = await asyncio.get_running_loop().run_in_executor(None, validate_url, url)
if url_error is not None:
log.warning('Rejected URL "%s": %s', url, url_error)
return {'status': 'error', 'msg': url_error}
try:
entry = await asyncio.get_running_loop().run_in_executor(
None,
+8 -8
View File
@@ -33,10 +33,10 @@
"@angular/platform-browser-dynamic": "^22.0.6",
"@angular/service-worker": "^22.0.6",
"@fortawesome/angular-fontawesome": "~4.0.0",
"@fortawesome/fontawesome-svg-core": "^7.3.0",
"@fortawesome/free-brands-svg-icons": "^7.3.0",
"@fortawesome/free-regular-svg-icons": "^7.3.0",
"@fortawesome/free-solid-svg-icons": "^7.3.0",
"@fortawesome/fontawesome-svg-core": "^7.3.1",
"@fortawesome/free-brands-svg-icons": "^7.3.1",
"@fortawesome/free-regular-svg-icons": "^7.3.1",
"@fortawesome/free-solid-svg-icons": "^7.3.1",
"@ng-bootstrap/ng-bootstrap": "^21.0.0",
"@ng-select/ng-select": "^23.2.0",
"@popperjs/core": "^2.11.8",
@@ -49,13 +49,13 @@
},
"devDependencies": {
"@angular-eslint/builder": "22.0.0",
"@angular/build": "^22.0.5",
"@angular/cli": "^22.0.5",
"@angular/build": "^22.0.7",
"@angular/cli": "^22.0.7",
"@angular/compiler-cli": "^22.0.6",
"@angular/localize": "^22.0.6",
"@eslint/js": "^9.39.4",
"@eslint/js": "^9.39.5",
"angular-eslint": "22.0.0",
"eslint": "^9.39.4",
"eslint": "^9.39.5",
"jsdom": "^27.4.0",
"typescript": "~6.0.3",
"typescript-eslint": "8.62.0",
+258 -285
View File
File diff suppressed because it is too large Load Diff
Generated
+10 -10
View File
@@ -106,14 +106,14 @@ wheels = [
[[package]]
name = "anyio"
version = "4.14.1"
version = "4.14.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "idna" },
]
sdist = { url = "https://files.pythonhosted.org/packages/3b/72/5562aabb8dd7181e8e860622a38bea08d17842b99ecd4c91f84ac95251b0/anyio-4.14.1.tar.gz", hash = "sha256:8d648a3544c1a700e3ff78615cd679e4c5c3f149904287e73687b2596963629e", size = 254831, upload-time = "2026-06-24T20:56:06.017Z" }
sdist = { url = "https://files.pythonhosted.org/packages/61/cc/a381afa6efea9f496eff839d4a6a1aed3bfafc7b3ab4b0d1b243a12573dd/anyio-4.14.2.tar.gz", hash = "sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f", size = 260176, upload-time = "2026-07-12T20:29:07.082Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/b0/7b/90df4a0a816d98d6ea26f559d87836d494a2cf1fcf063be67df50a7bcc30/anyio-4.14.1-py3-none-any.whl", hash = "sha256:4e5533c5b8ff0a24f5d7a176cbe6877129cd183893f66b537f8f227d10527d72", size = 124875, upload-time = "2026-06-24T20:56:04.413Z" },
{ url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" },
]
[[package]]
@@ -366,15 +366,15 @@ wheels = [
[[package]]
name = "deno"
version = "2.9.2"
version = "2.9.3"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/dc/3c/bf3a4e2c00f7eefd4ed2fd033a958cab3e92dcc539d3ab80351850cae99c/deno-2.9.2.tar.gz", hash = "sha256:3b77d7689c15fb2c47d5a3927dc8cf70dbd408bc92e20c70da079c6d78980e34", size = 8164, upload-time = "2026-07-08T14:38:54.267Z" }
sdist = { url = "https://files.pythonhosted.org/packages/98/ab/638749d76881f74d100a079414745f0a0fd20bba38e27ab4a3a6495d8264/deno-2.9.3.tar.gz", hash = "sha256:73268cbac7f7c4ff1983e49420a93f3f3f2f2e4e4372436f0942d04d02c9e943", size = 8166, upload-time = "2026-07-15T15:37:50.565Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/27/fe/4a02a256f19b3f31ac17ec37f6467ecf523601ed753e6c9177865dd2bbb2/deno-2.9.2-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:fb4bb5362d02c193f8086668789076f366416980b79277c06ea6cff73866b668", size = 42341183, upload-time = "2026-07-08T14:38:36.384Z" },
{ url = "https://files.pythonhosted.org/packages/5a/3e/235530ac9b9c206ee8c098e2b215f711a71d915d8d7d2be77599c8c1db4b/deno-2.9.2-py3-none-macosx_11_0_arm64.whl", hash = "sha256:f6f74fd8698a8a9a31af14eed5730f1964ab50eaa0129079d11d91fa92496f60", size = 37984508, upload-time = "2026-07-08T14:38:40.006Z" },
{ url = "https://files.pythonhosted.org/packages/7b/27/06a166fca538c06dfae4c45dbb92c2ebf3083341141966a3d649272ea734/deno-2.9.2-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:8b2fd9318d452b7e44c45968916670a0c5452d72796e1af9abfaa7a389e4aa6b", size = 42089854, upload-time = "2026-07-08T14:38:43.841Z" },
{ url = "https://files.pythonhosted.org/packages/09/b4/38338550a5ca0a7bcc39991e3d48e46ce3699413460dff32f383a611d822/deno-2.9.2-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:bf0471db7238e90810bca77a8d2c28646317b5a3fd9e5cdfdf73b251fde11b83", size = 43916921, upload-time = "2026-07-08T14:38:48.057Z" },
{ url = "https://files.pythonhosted.org/packages/a9/3d/c490c27e1f720baf014bb2c18b29d09b11c21d69fadfd69639e5904cdc95/deno-2.9.2-py3-none-win_amd64.whl", hash = "sha256:8e815d66b3e1314d028b2f0a05c75a331dd7025f4c825e12dd1f057207cee1f6", size = 41621982, upload-time = "2026-07-08T14:38:51.928Z" },
{ url = "https://files.pythonhosted.org/packages/07/fa/b4e1e2b3b894ed992c78a6796a43dc775edff895239d83c0e9a10457f4f7/deno-2.9.3-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:1deef9fe97ab28d5d4fea07693075e9ccc62f4ea97a653047a770beb01ecd307", size = 42354026, upload-time = "2026-07-15T15:37:34.977Z" },
{ url = "https://files.pythonhosted.org/packages/3d/cc/4090ef7370005b740efe57ec6ff75a3471393abe9816f1459e73dbbf0429/deno-2.9.3-py3-none-macosx_11_0_arm64.whl", hash = "sha256:78635cd9f0802ce7125aa3ba6feb5775460f55d432611374f16b454ffb8a1308", size = 37997909, upload-time = "2026-07-15T15:37:38.291Z" },
{ url = "https://files.pythonhosted.org/packages/14/07/70b7c965bbca5f3d158859acda0e90c041f1fa98b8c9ac8eff3fafd8b96c/deno-2.9.3-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:7c1c669827a480ef9e2710dd762560811da35dca04a9e90b23c0518587dadb8a", size = 42100873, upload-time = "2026-07-15T15:37:41.539Z" },
{ url = "https://files.pythonhosted.org/packages/7b/f5/0a07cc19a27476011719dd017a98950c7ccc33058c2ffc07019ba3ad9a5e/deno-2.9.3-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:3e724d5c8df13a90f6b50b16cc8df1acfe64c61a7736a774e11d61f3bca3a3b1", size = 43928967, upload-time = "2026-07-15T15:37:45.175Z" },
{ url = "https://files.pythonhosted.org/packages/e1/9a/030847bd4ea6cefbb471cae2b63c9ab29a3ab6c1809484447752bbcbcd52/deno-2.9.3-py3-none-win_amd64.whl", hash = "sha256:e2ebe2b5c1a7ee9daeaf8caf14ebc33fda96b4c2399da259e761fe2f82d22fa4", size = 41630201, upload-time = "2026-07-15T15:37:48.344Z" },
]
[[package]]