[core] Fix the byte string-format going over the specified byte limit

[nurupo]

IMPORTANT: PRs without the template will be CLOSED

Description of your pull request and other information

The byte string-format's purpose is to limit the formatting string to be no more than N bytes. The byte string-format should be applied after the sanitization is done, as sanitize might replace a single byte character with a multi-byte one, e.g. '/' with '⧸', making the resulting string go over the desired byte limit.

Fixes #10060

Template

Before submitting a pull request make sure you have:

• At least skimmed through contributing guidelines including yt-dlp coding conventions
• Searched the bugtracker for similar pull requests

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

• I am the original author of this code and I am willing to release it under Unlicense
• I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

• Fix or improvement to an extractor (Make sure to add/update tests)
• New extractor (Piracy websites will not be accepted)
• Core bug fix/improvement
• New feature (It is strongly recommended to open an issue first)