ERROR 500 - when using impersonate yt-dlp works directly but not via metube #559

admapop · 2024-12-27T11:57:36Z

YTDL_OPTIONS={"cookiefile":"/cookies/cookies.txt","verbose":"true","impersonate":"chrome"}

If I enter the docker container and execute yt-dlp directly with the cookiefile and impersonate, the download works.

When I try via the UI I get

INFO:ytdl:adding https://www.udemy.com/course/complete-react-developer-zero-to-mastery/learn/lecture/15090998#overview: quality='best' format='any' already=None folder=None custom_name_prefix='' playlist_strict_mode=False playlist_item_limit=0
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8 (No ANSI), error utf-8 (No ANSI), screen utf-8 (No ANSI)
[debug] yt-dlp version [email protected] from yt-dlp/yt-dlp [65cf46cdd] (pip) API
[debug] params: {'quiet': True, 'no_color': True, 'extract_flat': True, 'ignore_no_formats_error': True, 'noplaylist': False, 'paths': {'home': '/downloads', 'temp': '/downloads'}, 'cookiefile': '/cookies/cookies.txt', 'verbose': 'true', 'impersonate': 'chrome', 'color': 'no_color', 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}}
[debug] Python 3.11.11 (CPython aarch64 64bit) - Linux-6.8.0-1013-oracle-aarch64-with (OpenSSL 3.3.2 3 Sep 2024)
[debug] exe versions: ffmpeg 6.1.2 (setts), ffprobe 6.1.2
[debug] Optional libraries: Cryptodome-3.21.0, brotli-1.1.0, certifi-2024.08.30, curl_cffi-0.7.1, mutagen-1.47.0, requests-2.32.3, sqlite3-3.47.1, urllib3-2.2.3, websockets-13.1
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 477, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 559, in _handle
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/app/main.py", line 146, in add
    status = await dqueue.add(url, quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/app/ytdl.py", line 334, in add
    entry = await asyncio.get_running_loop().run_in_executor(None, self.__extract_info, url, playlist_strict_mode)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/app/ytdl.py", line 223, in __extract_info
    return yt_dlp.YoutubeDL(params={
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 742, in __init__
    if not self._impersonate_target_available(impersonate_target):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 4146, in _impersonate_target_available
    return any(
           ^^^^
  File "/usr/local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 4147, in <genexpr>
    rh.is_supported_target(target)
  File "/usr/local/lib/python3.11/site-packages/yt_dlp/networking/impersonate.py", line 119, in is_supported_target
    assert isinstance(target, ImpersonateTarget)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
INFO:aiohttp.access:10.0.7.1 [27/Dec/2024:11:50:53 +0000] "POST /add HTTP/1.1" 500 346 "https://MYDOMAIN/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
INFO:ytdl:adding https://www.udemy.com/course/complete-react-developer-zero-to-mastery/learn/lecture/15090998#overview: quality='best' format='mp4' already=None folder=None custom_name_prefix='' playlist_strict_mode=False playlist_item_limit=0
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8 (No ANSI), error utf-8 (No ANSI), screen utf-8 (No ANSI)
[debug] yt-dlp version [email protected] from yt-dlp/yt-dlp [65cf46cdd] (pip) API
[debug] params: {'quiet': True, 'no_color': True, 'extract_flat': True, 'ignore_no_formats_error': True, 'noplaylist': False, 'paths': {'home': '/downloads', 'temp': '/downloads'}, 'cookiefile': '/cookies/cookies.txt', 'verbose': 'true', 'impersonate': 'safari', 'color': 'no_color', 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}}
[debug] Python 3.11.11 (CPython aarch64 64bit) - Linux-6.8.0-1013-oracle-aarch64-with (OpenSSL 3.3.2 3 Sep 2024)
[debug] exe versions: ffmpeg 6.1.2 (setts), ffprobe 6.1.2
[debug] Optional libraries: Cryptodome-3.21.0, brotli-1.1.0, certifi-2024.08.30, curl_cffi-0.7.1, mutagen-1.47.0, requests-2.32.3, sqlite3-3.47.1, urllib3-2.2.3, websockets-13.1
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 477, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 559, in _handle
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/app/main.py", line 146, in add
    status = await dqueue.add(url, quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/app/ytdl.py", line 334, in add
    entry = await asyncio.get_running_loop().run_in_executor(None, self.__extract_info, url, playlist_strict_mode)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/app/ytdl.py", line 223, in __extract_info
    return yt_dlp.YoutubeDL(params={
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 742, in __init__
    if not self._impersonate_target_available(impersonate_target):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 4146, in _impersonate_target_available
    return any(
           ^^^^
  File "/usr/local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 4147, in <genexpr>
    rh.is_supported_target(target)
  File "/usr/local/lib/python3.11/site-packages/yt_dlp/networking/impersonate.py", line 119, in is_supported_target
    assert isinstance(target, ImpersonateTarget)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
INFO:aiohttp.access:10.0.7.1 [27/Dec/2024:11:50:57 +0000] "POST /add HTTP/1.1" 500 346 "https://MYDOMAIN/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

The text was updated successfully, but these errors were encountered:

PikuZheng · 2024-12-27T12:49:29Z

Have you inspected into the container and installed the chrome component?

admapop · 2024-12-27T14:18:28Z

@PikuZheng how do you mean?

I've opened a sh in the container and was able to use yt-dlp directly with impersonate chrome, but didn't do much else.

I also have a custom dockerfile to install some dependencies

PikuZheng · 2024-12-27T23:43:12Z

[debug] yt-dlp version [email protected] from yt-dlp/yt-dlp [65cf46cdd] (pip) API
[debug] params: {'quiet': True, 'no_color': True, 'extract_flat': True, 'ignore_no_formats_error': True, 'noplaylist': False, 'paths': {'home': '/downloads', 'temp': '/downloads'}, 'cookiefile': '/cookies/cookies.txt', 'verbose': 'true', 'impersonate': 'chrome', 'color': 'no_color', 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}}
[debug] Python 3.11.11 (CPython aarch64 64bit) - Linux-6.8.0-1013-oracle-aarch64-with (OpenSSL 3.3.2 3 Sep 2024)
[debug] exe versions: ffmpeg 6.1.2 (setts), ffprobe 6.1.2
[debug] Optional libraries: Cryptodome-3.21.0, brotli-1.1.0, certifi-2024.08.30, curl_cffi-0.7.1, mutagen-1.47.0, requests-2.32.3, sqlite3-3.47.1, urllib3-2.2.3, websockets-13.1
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi

There is no impersonate named chrome, so I think something is missing in the runtime environment.
On the other hand, if the command line works fine in the same environment but the embedded one doesn't, it may be because the ytdl_options are incorrect. Can you write the command line that works properly so that we can determine the issue?

admapop · 2024-12-28T10:56:10Z

[debug] yt-dlp version [email protected] from yt-dlp/yt-dlp [65cf46cdd] (pip) API
[debug] params: {'quiet': True, 'no_color': True, 'extract_flat': True, 'ignore_no_formats_error': True, 'noplaylist': False, 'paths': {'home': '/downloads', 'temp': '/downloads'}, 'cookiefile': '/cookies/cookies.txt', 'verbose': 'true', 'impersonate': 'chrome', 'color': 'no_color', 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}}
[debug] Python 3.11.11 (CPython aarch64 64bit) - Linux-6.8.0-1013-oracle-aarch64-with (OpenSSL 3.3.2 3 Sep 2024)
[debug] exe versions: ffmpeg 6.1.2 (setts), ffprobe 6.1.2
[debug] Optional libraries: Cryptodome-3.21.0, brotli-1.1.0, certifi-2024.08.30, curl_cffi-0.7.1, mutagen-1.47.0, requests-2.32.3, sqlite3-3.47.1, urllib3-2.2.3, websockets-13.1
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi

There is no impersonate named chrome, so I think something is missing in the runtime environment. On the other hand, if the command line works fine in the same environment but the embedded one doesn't, it may be because the ytdl_options are incorrect. Can you write the command line that works properly so that we can determine the issue?

You might be right about chrome not being right, i just saw in the code that there's a quick mapping between chrome and chrome-110.

This is the command I use that works

yt-dlp https://www.udemy.com/course/complete-react-developer-zero-to-mastery/learn/lecture/14754858#overview --cookies /cookies/cookies.txt --impersonate="chrome-110"

My YTDL options are like this

YTDL_OPTIONS={"cookiefile":"/cookies/cookies.txt","verbose":"true","impersonate":"chrome"}

And loaded in the docker compose via env var

    environment:
      - SERVICE_FQDN_METUBE_8081
      - UID=1000
      - GID=1000
      - YTDL_OPTIONS

PikuZheng · 2024-12-28T14:38:30Z

YTDL_OPTIONS={"cookiefile":"/cookies/cookies.txt","verbose":"true","impersonate":"chrome"}

according to https://github.com/yt-dlp/yt-dlp/blob/0b6b7742c2e7f2a1fcb0b54ef3dd484bab404b3f/yt_dlp/YoutubeDL.py#L422
impersonate should be an ImpersonateTarget object. metube processes ytdl_options as json, which does not support this

or you need to modify YoutubeDL.py so that json is converted to class object

admapop · 2024-12-28T15:06:42Z

YTDL_OPTIONS={"cookiefile":"/cookies/cookies.txt","verbose":"true","impersonate":"chrome"}

according to https://github.com/yt-dlp/yt-dlp/blob/0b6b7742c2e7f2a1fcb0b54ef3dd484bab404b3f/yt_dlp/YoutubeDL.py#L422 impersonate should be an ImpersonateTarget object. metube processes ytdl_options as json, which does not support this

or you need to modify YoutubeDL.py so that json is converted to class object

Good point. I made a quick PR to create the ImpersonateTarget object if the impersonate property exists

#560

PikuZheng · 2024-12-28T15:19:09Z

nice job

PikuZheng · 2024-12-28T15:28:32Z

Depending on your pr, should ytdl_options be written as "impersonate": "chrome-110:windows-10" ?

admapop · 2024-12-28T18:42:48Z

Depending on your pr, should ytdl_options be written as "impersonate": "chrome-110:windows-10" ?

I'm not sure? When running the command directly I can get away with using chrome-110 without specifying OS.

PikuZheng · 2024-12-28T23:49:50Z

What I tried

YTDL_OPTIONS={ 
  "impersonate": {
    "client": "chrome",
    "version": "110",
    "os": "windows",
    "os_version": "10"
  }
}

main.py

from yt_dlp.networking.impersonate import ImpersonateTarget
class Config:
    def __init__(self):
        ................
        if impersonate_data := self.YTDL_OPTIONS.get("impersonate"):
            self.YTDL_OPTIONS["impersonate"] = ImpersonateTarget(**impersonate_data)

admapop mentioned this issue Dec 28, 2024

fix: impersonate property needs to be of the right type and not as it is being passed now via JSON #560

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ERROR 500 - when using impersonate yt-dlp works directly but not via metube #559

ERROR 500 - when using impersonate yt-dlp works directly but not via metube #559

admapop commented Dec 27, 2024

PikuZheng commented Dec 27, 2024

admapop commented Dec 27, 2024

PikuZheng commented Dec 27, 2024

admapop commented Dec 28, 2024

PikuZheng commented Dec 28, 2024

admapop commented Dec 28, 2024

PikuZheng commented Dec 28, 2024

PikuZheng commented Dec 28, 2024

admapop commented Dec 28, 2024

PikuZheng commented Dec 28, 2024

ERROR 500 - when using impersonate yt-dlp works directly but not via metube #559

ERROR 500 - when using impersonate yt-dlp works directly but not via metube #559

Comments

admapop commented Dec 27, 2024

PikuZheng commented Dec 27, 2024

admapop commented Dec 27, 2024

PikuZheng commented Dec 27, 2024

admapop commented Dec 28, 2024

PikuZheng commented Dec 28, 2024

admapop commented Dec 28, 2024

PikuZheng commented Dec 28, 2024

PikuZheng commented Dec 28, 2024

admapop commented Dec 28, 2024

PikuZheng commented Dec 28, 2024