Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TVer] Trying to download from this site gives a time out on brightcove's part #10199

Closed
10 of 11 tasks
GT2000byte opened this issue Jun 17, 2024 · 9 comments · Fixed by #10202
Closed
10 of 11 tasks

[TVer] Trying to download from this site gives a time out on brightcove's part #10199

GT2000byte opened this issue Jun 17, 2024 · 9 comments · Fixed by #10202
Labels
patch-available There is patch available that should fix this issue. Someone needs to make a PR with it site-bug Issue with a specific website

Comments

@GT2000byte
Copy link

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

  • I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

Region

Japan

Provide a description that is worded well enough to be understood

I've been trying to download a show from this site for some time. The problem seems to be with brightcove and giving time out errors whenever trying to doing a download, the weird thing is the video does play correctly on my browser. I've also tried other VPN services in case the one I regularly use gives the problem, but it still refuses to download said video.

Provide verbose output that clearly demonstrates the problem

  • Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
  • If using API, add 'verbose': True to YoutubeDL params instead
  • Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['-vU', 'https://tver.jp/episodes/ep063fuyyp']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version [email protected] from yt-dlp/yt-dlp [12b248ce6] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg 6.1-full_build-www.gyan.dev (setts), ffprobe 6.1-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, curl_cffi-0.5.10, mutagen-1.47.0, requests-2.32.2, sqlite3-3.35.5, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Loaded 1820 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: [email protected] from yt-dlp/yt-dlp
yt-dlp is up to date ([email protected] from yt-dlp/yt-dlp)
[TVer] Creating session
[TVer] Extracting URL: https://tver.jp/episodes/ep063fuyyp
[TVer] ep063fuyyp: Downloading JSON metadata
[TVer] ep063fuyyp: Downloading JSON metadata
[brightcove:new] Extracting URL: http://players.brightcove.net/4394098882001/default_default/index.html?videoId=ref:c741b3554c65468c832ff3457d8c74e4#__youtubedl_smuggle=%7B%22geo_countries%22%3A+%5B%22JP%22%5D%7D
[debug] Using fake IP 133.12.161.121 (JP) as X-Forwarded-For
[brightcove:new] ref:c741b3554c65468c832ff3457d8c74e4: Downloading JSON metadata
WARNING: [brightcove:new] Unable to download JSON metadata: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)
[brightcove:new] ref:c741b3554c65468c832ff3457d8c74e4: Downloading webpage
ERROR: [brightcove:new] Unable to download webpage: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0) (caused by TransportError("HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)"))
  File "yt_dlp\extractor\common.py", line 734, in extract
  File "yt_dlp\extractor\brightcove.py", line 911, in _real_extract
  File "yt_dlp\extractor\brightcove.py", line 880, in extract_policy_key
  File "yt_dlp\extractor\common.py", line 1182, in _download_webpage
  File "yt_dlp\extractor\common.py", line 1133, in download_content
  File "yt_dlp\extractor\adobepass.py", line 1366, in _download_webpage_handle
  File "yt_dlp\extractor\common.py", line 954, in _download_webpage_handle
  File "yt_dlp\extractor\common.py", line 903, in _request_webpage

  File "urllib3\connectionpool.py", line 537, in _make_request
  File "urllib3\connection.py", line 466, in getresponse
  File "http\client.py", line 1344, in getresponse
  File "http\client.py", line 307, in begin
  File "http\client.py", line 268, in _read_status
  File "socket.py", line 669, in readinto
socket.timeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "requests\adapters.py", line 589, in send
  File "urllib3\connectionpool.py", line 847, in urlopen
  File "urllib3\util\retry.py", line 445, in increment
  File "urllib3\util\util.py", line 39, in reraise
  File "urllib3\connectionpool.py", line 793, in urlopen
  File "urllib3\connectionpool.py", line 539, in _make_request
  File "urllib3\connectionpool.py", line 370, in _raise_timeout
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "yt_dlp\networking\_requests.py", line 329, in _send
  File "requests\sessions.py", line 589, in request
  File "requests\sessions.py", line 703, in send
  File "requests\adapters.py", line 635, in send
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "yt_dlp\extractor\common.py", line 890, in _request_webpage
  File "yt_dlp\YoutubeDL.py", line 4142, in urlopen
  File "yt_dlp\networking\common.py", line 117, in send
  File "yt_dlp\networking\_helper.py", line 208, in wrapper
  File "yt_dlp\networking\common.py", line 337, in send
  File "yt_dlp\networking\_requests.py", line 353, in _send
yt_dlp.networking.exceptions.TransportError: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)
@GT2000byte GT2000byte added site-bug Issue with a specific website triage Untriaged issue labels Jun 17, 2024
@bashonly
Copy link
Member

bashonly commented Jun 17, 2024

Could you try running this:

yt-dlp -v "https://players.brightcove.net/4394098882001/default_default/index.html?videoId=ref:c741b3554c65468c832ff3457d8c74e4#__youtubedl_smuggle=%7B%22geo_countries%22%3A+%5B%22JP%22%5D%7D"

@GT2000byte
Copy link
Author

GT2000byte commented Jun 17, 2024

I did just run the line you mentioned, but it's giving me the same time out error. I could drop in the verbose log again if you would like to take a look.

@bashonly
Copy link
Member

bashonly commented Jun 17, 2024

Sure, I'll always take a verbose log.

How about does this one work for you:

yt-dlp -v "https://players.brightcove.net/4394098882001/default_default/index.html?videoId=ref:c741b3554c65468c832ff3457d8c74e4"

You mentioned you are using a VPN. What if you don't use any VPN/proxy?

@GT2000byte
Copy link
Author

I'll give you the verbose of the first line you recommended:

`[debug] Command-line config: ['-vU', 'yt-dlp', '-v', 'https://players.brightcove.net/4394098882001/default_default/index.html?videoId=ref:c741b3554c65468c832ff3457d8c74e4#__youtubedl_smuggle=%7B%22geo_countries%22%3A+%5B%22JP%22%5D%7D']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version [email protected] from yt-dlp/yt-dlp [12b248c] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k 25 Mar 2021)
[debug] exe versions: ffmpeg 6.1-full_build-www.gyan.dev (setts), ffprobe 6.1-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, curl_cffi-0.5.10, mutagen-1.47.0, requests-2.32.2, sqlite3-3.35.5, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Loaded 1820 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: [email protected] from yt-dlp/yt-dlp
yt-dlp is up to date ([email protected] from yt-dlp/yt-dlp)
[CommonMistakes] Extracting URL: yt-dlp
ERROR: [CommonMistakes] You've asked yt-dlp to download the URL "yt-dlp". That doesn't make any sense. Simply remove the parameter in your command or configuration.
File "yt_dlp\extractor\common.py", line 734, in extract
File "yt_dlp\extractor\commonmistakes.py", line 25, in _real_extract

[brightcove:new] Extracting URL: https://players.brightcove.net/4394098882001/default_default/index.html?videoId=ref:c741b3554c65468c832ff3457d8c74e4#__youtubedl_smuggle=%7B%22geo_countries%22%3A+%5B%22JP%22%5D%7D
[debug] Using fake IP 133.157.22.141 (JP) as X-Forwarded-For
[brightcove:new] ref:c741b3554c65468c832ff3457d8c74e4: Downloading JSON metadata
WARNING: [brightcove:new] Unable to download JSON metadata: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)
[brightcove:new] ref:c741b3554c65468c832ff3457d8c74e4: Downloading webpage
ERROR: [brightcove:new] Unable to download webpage: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0) (caused by TransportError("HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)"))
File "yt_dlp\extractor\common.py", line 734, in extract
File "yt_dlp\extractor\brightcove.py", line 911, in _real_extract
File "yt_dlp\extractor\brightcove.py", line 880, in extract_policy_key
File "yt_dlp\extractor\common.py", line 1182, in _download_webpage
File "yt_dlp\extractor\common.py", line 1133, in download_content
File "yt_dlp\extractor\adobepass.py", line 1366, in _download_webpage_handle
File "yt_dlp\extractor\common.py", line 954, in _download_webpage_handle
File "yt_dlp\extractor\common.py", line 903, in _request_webpage

File "urllib3\connectionpool.py", line 537, in _make_request
File "urllib3\connection.py", line 466, in getresponse
File "http\client.py", line 1344, in getresponse
File "http\client.py", line 307, in begin
File "http\client.py", line 268, in _read_status
File "socket.py", line 669, in readinto
socket.timeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "requests\adapters.py", line 589, in send
File "urllib3\connectionpool.py", line 847, in urlopen
File "urllib3\util\retry.py", line 445, in increment
File "urllib3\util\util.py", line 39, in reraise
File "urllib3\connectionpool.py", line 793, in urlopen
File "urllib3\connectionpool.py", line 539, in _make_request
File "urllib3\connectionpool.py", line 370, in _raise_timeout
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "yt_dlp\networking_requests.py", line 329, in _send
File "requests\sessions.py", line 589, in request
File "requests\sessions.py", line 703, in send
File "requests\adapters.py", line 635, in send
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "yt_dlp\extractor\common.py", line 890, in _request_webpage
File "yt_dlp\YoutubeDL.py", line 4142, in urlopen
File "yt_dlp\networking\common.py", line 117, in send
File "yt_dlp\networking_helper.py", line 208, in wrapper
File "yt_dlp\networking\common.py", line 337, in send
File "yt_dlp\networking_requests.py", line 353, in _send
yt_dlp.networking.exceptions.TransportError: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)`

And here's the verbose of the second line:

`[debug] Command-line config: ['-v', 'https://players.brightcove.net/4394098882001/default_default/index.html?videoId=ref:c741b3554c65468c832ff3457d8c74e4']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version [email protected] from yt-dlp/yt-dlp [12b248c] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k 25 Mar 2021)
[debug] exe versions: ffmpeg 6.1-full_build-www.gyan.dev (setts), ffprobe 6.1-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, curl_cffi-0.5.10, mutagen-1.47.0, requests-2.32.2, sqlite3-3.35.5, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Loaded 1820 extractors
[brightcove:new] Extracting URL: https://players.brightcove.net/4394098882001/default_default/index.html?videoId=ref:c741b3554c65468c832ff3457d8c74e4
[brightcove:new] ref:c741b3554c65468c832ff3457d8c74e4: Downloading JSON metadata
WARNING: [brightcove:new] Unable to download JSON metadata: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)
[brightcove:new] ref:c741b3554c65468c832ff3457d8c74e4: Downloading webpage
ERROR: [brightcove:new] Unable to download webpage: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0) (caused by TransportError("HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)"))
File "yt_dlp\extractor\common.py", line 734, in extract
File "yt_dlp\extractor\brightcove.py", line 911, in _real_extract
File "yt_dlp\extractor\brightcove.py", line 880, in extract_policy_key
File "yt_dlp\extractor\common.py", line 1182, in _download_webpage
File "yt_dlp\extractor\common.py", line 1133, in download_content
File "yt_dlp\extractor\adobepass.py", line 1366, in _download_webpage_handle
File "yt_dlp\extractor\common.py", line 954, in _download_webpage_handle
File "yt_dlp\extractor\common.py", line 903, in _request_webpage

File "urllib3\connectionpool.py", line 537, in _make_request
File "urllib3\connection.py", line 466, in getresponse
File "http\client.py", line 1344, in getresponse
File "http\client.py", line 307, in begin
File "http\client.py", line 268, in _read_status
File "socket.py", line 669, in readinto
socket.timeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "requests\adapters.py", line 589, in send
File "urllib3\connectionpool.py", line 847, in urlopen
File "urllib3\util\retry.py", line 445, in increment
File "urllib3\util\util.py", line 39, in reraise
File "urllib3\connectionpool.py", line 793, in urlopen
File "urllib3\connectionpool.py", line 539, in _make_request
File "urllib3\connectionpool.py", line 370, in _raise_timeout
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "yt_dlp\networking_requests.py", line 329, in _send
File "requests\sessions.py", line 589, in request
File "requests\sessions.py", line 703, in send
File "requests\adapters.py", line 635, in send
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "yt_dlp\extractor\common.py", line 890, in _request_webpage
File "yt_dlp\YoutubeDL.py", line 4142, in urlopen
File "yt_dlp\networking\common.py", line 117, in send
File "yt_dlp\networking_helper.py", line 208, in wrapper
File "yt_dlp\networking\common.py", line 337, in send
File "yt_dlp\networking_requests.py", line 353, in _send
yt_dlp.networking.exceptions.TransportError: HTTPConnectionPool(host='players.brightcove.net', port=80): Read timed out. (read timeout=20.0)`

There doesn't seem to be a difference, is it?

You mentioned you are using a VPN. What if you don't use any VPN/proxy?

yt-dlp straight up won't work, it'll say the site is geo locked if I don't use a VPN in Japan.

@bashonly
Copy link
Member

@GT2000byte if you want to try one more thing, you could do this:

  1. Update yt-dlp to a patched pre-release I just published:
yt-dlp --update-to bashonly/yt-dlp@brightcove
  1. With the patched version, try the original URL again:
yt-dlp -v "https://tver.jp/episodes/ep063fuyyp"
  1. If it doesn't work, then share the verbose log again (please), and you can run this to update back to the stable release channel:
yt-dlp --update-to stable@latest

@GT2000byte
Copy link
Author

Okay, running the patched pre-release worked, the video is downloading just like it used to. It seems like said patch fixed whatever error is happening with the current stable release.

Thank you!

@bashonly bashonly added patch-available There is patch available that should fix this issue. Someone needs to make a PR with it and removed triage Untriaged issue labels Jun 17, 2024
@bashonly
Copy link
Member

What the patch did was upgrade all of the Brightcove extractor's requests to HTTPS:

diff --git a/yt_dlp/extractor/brightcove.py b/yt_dlp/extractor/brightcove.py
index dc0c83572..56d74764f 100644
--- a/yt_dlp/extractor/brightcove.py
+++ b/yt_dlp/extractor/brightcove.py
@@ -386,7 +386,7 @@ def _build_brightcove_url_from_js(cls, object_js):
     @classmethod
     def _make_brightcove_url(cls, params):
         return update_url_query(
-            'http://c.brightcove.com/services/viewer/htmlFederated', params)
+            'https://c.brightcove.com/services/viewer/htmlFederated', params)
 
     @classmethod
     def _extract_brightcove_url(cls, webpage):
@@ -470,7 +470,7 @@ def _real_extract(self, url):
                         if referer:
                             headers['Referer'] = referer
                         player_page = self._download_webpage(
-                            'http://link.brightcove.com/services/player/bcpid' + player_id[0],
+                            'https://link.brightcove.com/services/player/bcpid' + player_id[0],
                             video_id, headers=headers, fatal=False)
                         if player_page:
                             player_key = self._search_regex(
@@ -480,7 +480,7 @@ def _real_extract(self, url):
                     enc_pub_id = player_key.split(',')[1].replace('~', '=')
                     publisher_id = struct.unpack('>Q', base64.urlsafe_b64decode(enc_pub_id))[0]
             if publisher_id:
-                brightcove_new_url = f'http://players.brightcove.net/{publisher_id}/default_default/index.html?videoId={video_id}'
+                brightcove_new_url = f'https://players.brightcove.net/{publisher_id}/default_default/index.html?videoId={video_id}'
                 if referer:
                     brightcove_new_url = smuggle_url(brightcove_new_url, {'referrer': referer})
                 return self.url_result(brightcove_new_url, BrightcoveNewIE.ie_key(), video_id)
@@ -801,7 +801,7 @@ def _extract_brightcove_urls(ie, webpage):
         # Look for iframe embeds [1]
         for _, url in re.findall(
                 r'<iframe[^>]+src=(["\'])((?:https?:)?//players\.brightcove\.net/\d+/[^/]+/index\.html.+?)\1', webpage):
-            entries.append(url if url.startswith('http') else 'http:' + url)
+            entries.append(url if url.startswith(('http:', 'https:')) else 'https:' + url)
 
         # Look for <video> tags [2] and embed_in_page embeds [3]
         # [2] looks like:
@@ -830,7 +830,7 @@ def _extract_brightcove_urls(ie, webpage):
             player_id = player_id or attrs.get('data-player') or 'default'
             embed = embed or attrs.get('data-embed') or 'default'
 
-            bc_url = f'http://players.brightcove.net/{account_id}/{player_id}_{embed}/index.html?videoId={video_id}'
+            bc_url = f'https://players.brightcove.net/{account_id}/{player_id}_{embed}/index.html?videoId={video_id}'
 
             # Some brightcove videos may be embedded with video tag only and
             # without script tag or any mentioning of brightcove at all. Such
@@ -867,7 +867,7 @@ def _real_extract(self, url):
         store_pk = lambda x: self.cache.store('brightcove', policy_key_id, x)
 
         def extract_policy_key():
-            base_url = f'http://players.brightcove.net/{account_id}/{player_id}_{embed}/'
+            base_url = f'https://players.brightcove.net/{account_id}/{player_id}_{embed}/'
             config = self._download_json(
                 base_url + 'config.json', video_id, fatal=False) or {}
             policy_key = try_get(

@bashonly
Copy link
Member

Nightly has the patch now:

yt-dlp --update-to nightly

@GT2000byte
Copy link
Author

Thank you. I'll probably update yt-dlp to the Nightly releases for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
patch-available There is patch available that should fix this issue. Someone needs to make a PR with it site-bug Issue with a specific website
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants