fetch missing SSL certificates

[milahu]

fix #4469

based on danilobellini/aia#3

[CLAassistant]

All committers have signed the CLA.

[pep8speaks]

Hello @milahu! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file src/pyload/core/network/http/aia_retry_wrap_download.py:

Line 121:89: E501 line too long (96 > 88 characters)

We recommend to use black to automatically correct these files.

Comment last updated at 2024-06-16 09:25:18 UTC

[milahu]

mmkay. works for me

tests

• tests/test_ssl_certs.py based on aia/test/test.py
• https://ddownload.com/ urls

missing root certs are rare, so this is not-yet implemented in the webui
when pyload finds a missing root cert, the cert is saved to
~/.pyload2/cache/untrusted-root-certs/
to trust that root cert, it must be moved to
~/.pyload2/settings/trusted-root-certs/
and pyload must be restarted

todo: remove debug prints

this error is probably not caused by my changes
the request succeeds on retry

  File "pyload/core/network/http/http_request.py", line 236, in set_request_context
    self.c.setopt(pycurl.URL, url)
pycurl.error: cannot invoke setopt() - perform() is currently running

[2024-06-05 18:51:15]  DEBUG               pyload  ADDON AntiCaptcha: LOAD URL https://api.anti-captcha.com/createTask | get={} | post={"clientKey": "xxx", "softId": 976, "task": xxx | ref=True | cookies=True | just_header=False | decode=True | multipart=False | redirect=True | req=None
Exception in thread Thread-46:
Traceback (most recent call last):
  File "/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/core/threads/addon_thread.py", line 61, in run
    self.f(*self.args, **self.kwargs)
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/plugins/addons/AntiCaptcha.py", line 96, in _process_captcha
    api_data = self.api_request(
               ^^^^^^^^^^^^^^^^^
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/plugins/addons/AntiCaptcha.py", line 65, in api_request
    json_data = self.load(self.API_URL + method, post=json.dumps(post))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/plugins/base/plugin.py", line 217, in load
    html = req.load(
           ^^^^^^^^^
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/core/network/browser.py", line 128, in load
    return self.http.load(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/core/network/http/aia_retry_wrap_download.py", line 27, in download_wrapper_aia_retry
    return download(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/core/network/http/http_request.py", line 307, in load
    self.set_request_context(url, get, post, referer, cookies, multipart, decode)
  File "/nix/store/qi6lvjl6i1hiwp9r5fbinnia0nwj3rap-pyload-0.5.0b3.dev82-unstable/lib/python3.11/site-packages/pyload/core/network/http/http_request.py", line 236, in set_request_context
    self.c.setopt(pycurl.URL, url)
pycurl.error: cannot invoke setopt() - perform() is currently running

[vuolter]

Hi @milahu thanks for the pr!
I see some commits not related to the ssl issue you targeted, could you pls check them all again? thx.

[milahu]

done. should i squash this into one commit?

there are still some debug prints
which should be removed or converted to log.debug calls ...
feel free to help ^^
currently im more worried about fixing #3899