Release 2021.07.07

[peertube] only call description endpoint if necessary
https://github.com/ytdl-org/youtube-dl/commit/a8035827177d6b59aca03bd717acb6a9bdd75ada

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.23. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.23**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -42,9 +42,9 @@ Provide the complete verbose output of yt-dlp that clearly demonstrates the prob
 Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
  [debug] System config: []
  [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
+ [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.06.09
+ [debug] yt-dlp version 2021.06.23
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.23. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.23**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.23. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.23**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.23. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.23**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -44,9 +44,9 @@ Provide the complete verbose output of yt-dlp that clearly demonstrates the prob
 Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
  [debug] System config: []
  [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
+ [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.06.09
+ [debug] yt-dlp version 2021.06.23
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.23. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.23**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE_tmpl/1_broken_site.md

@@ -42,7 +42,7 @@ Provide the complete verbose output of yt-dlp that clearly demonstrates the prob
 Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
  [debug] System config: []
  [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
+ [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
  [debug] yt-dlp version %(version)s
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2

.github/ISSUE_TEMPLATE_tmpl/4_bug_report.md

@@ -44,7 +44,7 @@ Provide the complete verbose output of yt-dlp that clearly demonstrates the prob
 Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
  [debug] System config: []
  [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
+ [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
  [debug] yt-dlp version %(version)s
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2

CONTRIBUTING.md

@@ -3,7 +3,7 @@
 $ youtube-dl -v <your command line>
 [debug] System config: []
 [debug] User config: []
-[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=BaW_jenozKcj']
+[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=BaW_jenozKc']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] youtube-dl version 2015.12.06
 [debug] Git HEAD: 135392e

CONTRIBUTORS

@@ -52,5 +52,9 @@ hhirtz
 louie-github
 MinePlayersPE
 olifre
-rhsmachine
+rhsmachine/zenerdi0de
 nihil-admirari
+krichbanana
+ohmybahgosh
+nyuszika7h
+blackjack4494

Changelog.md

@@ -19,12 +19,64 @@
 -->
 
 
+### 2021.07.07
+
+* Merge youtube-dl: Upto [commit/a803582](https://github.com/ytdl-org/youtube-dl/commit/a8035827177d6b59aca03bd717acb6a9bdd75ada)
+* Add `--extractor-args` to pass extractor-specific arguments
+    * Add extractor option `skip` for `youtube`. Eg: `--extractor-args youtube:skip=hls,dash`
+    * Deprecates --youtube-skip-dash-manifest, --youtube-skip-hls-manifest, --youtube-include-dash-manifest, --youtube-include-hls-manifest
+* Allow `--list...` options to work with `--print`, `--quiet` and other `--list...` options
+* [youtube] Use `player` API for additional video extraction requests by [colethedj](https://github.com/colethedj)
+    * **Fixes youtube premium music** (format 141) extraction
+    * Adds extractor option `player_client` = `web`/`android`
+        * **`--extractor-args youtube:player_client=android` works around the throttling** for the time-being
+    * Adds extractor option `player_skip=config`
+    * Adds age-gate fallback using embedded client
+* [youtube] Choose correct Live chat API for upcoming streams by [krichbanana](https://github.com/krichbanana)
+* [youtube] Fix subtitle names for age-gated videos
+* [youtube:comments] Fix error handling and add `itct` to params by [colethedj](https://github.com/colethedj)
+* [youtube_live_chat] Fix download with cookies by [siikamiika](https://github.com/siikamiika)
+* [youtube_live_chat] use `clickTrackingParams` by [siikamiika](https://github.com/siikamiika)
+* [Funimation] Rewrite extractor
+    * Add `FunimationShowIE` by [Mevious](https://github.com/Mevious)
+    * **Treat the different versions of an episode as different formats of a single video**
+        * This changes the video `id` and will break break existing archives
+        * Compat option `seperate-video-versions` to fall back to old behavior including using the old video ids
+    * Support direct `/player/` URL
+    * Extractor options `language` and `version` to pre-select them during extraction
+        * These options may be removed in the future if we can extract all formats without additional network requests
+        * Do not rely on these for format selection and use `-f` filters instead
+* [AdobePass] Add Spectrum MSO by [kevinoconnor7](https://github.com/kevinoconnor7), [ohmybahgosh](https://github.com/ohmybahgosh)
+* [facebook] Extract description and fix title
+* [fancode] Fix extraction, support live and allow login with refresh token by [zenerdi0de](https://github.com/zenerdi0de)
+* [plutotv] Improve `_VALID_URL`
+* [RCTIPlus] Add extractor by [MinePlayersPE](https://github.com/MinePlayersPE)
+* [Soundcloud] Allow login using oauth token by [blackjack4494](https://github.com/blackjack4494)
+* [TBS] Support livestreams by [llacb47](https://github.com/llacb47)
+* [videa] Fix extraction by [nyuszika7h](https://github.com/nyuszika7h)
+* [yahoo] Fix extraction by [llacb47](https://github.com/llacb47), [pukkandan](https://github.com/pukkandan)
+* Process videos when using `--ignore-no-formats-error` by [krichbanana](https://github.com/krichbanana)
+* Fix `--throttled-rate` when using `--load-info-json`
+* Fix `--flat-playlist` when entry has no `ie_key`
+* Fix `check_formats` catching `ExtractorError` instead of `DownloadError`
+* Fix deprecated option `--list-formats-old`
+* [downloader/ffmpeg] Fix `--ppa` when using simultaneous download
+* [extractor] Prevent unnecessary download of hls manifests and refactor `hls_split_discontinuity`
+* [fragment] Handle status of download and errors in threads correctly; and minor refactoring
+* [thumbnailsconvertor] Treat `jpeg` as `jpg`
+* [utils] Fix issues with `LazyList` reversal
+* [extractor] Allow extractors to set their own login hint
+* [cleanup] Simplify format selector code with `LazyList` and `yield from`
+* [cleanup] Clean `extractor.common._merge_subtitles` signature
+* [cleanup] Fix some typos
+
+
 ### 2021.06.23
 
 * Merge youtube-dl: Upto [commit/379f52a](https://github.com/ytdl-org/youtube-dl/commit/379f52a4954013767219d25099cce9e0f9401961)
 * **Add option `--throttled-rate`** below which video data is re-extracted
 * [fragment] **Merge during download for `-N`**, and refactor `hls`/`dash`
-* [websockets] Add `WebSocketFragmentFD`by [nao20010128nao](https://github.com/nao20010128nao), [pukkandan](https://github.com/pukkandan)
+* [websockets] Add `WebSocketFragmentFD` by [nao20010128nao](https://github.com/nao20010128nao), [pukkandan](https://github.com/pukkandan)
 * Allow `images` formats in addition to video/audio
 * [downloader/mhtml] Add new downloader for slideshows/storyboards by [fstirlitz](https://github.com/fstirlitz)
 * [youtube] Temporary **fix for age-gate**

README.md

@@ -53,6 +53,7 @@ yt-dlp is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork based on t
     * [Format Selection examples](#format-selection-examples)
 * [MODIFYING METADATA](#modifying-metadata)
     * [Modifying metadata examples](#modifying-metadata-examples)
+* [EXTRACTOR ARGUMENTS](#extractor-arguments)
 * [PLUGINS](#plugins)
 * [DEPRECATED OPTIONS](#deprecated-options)
 * [MORE](#more)
@@ -84,9 +85,9 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **Aria2c with HLS/DASH**: You can use `aria2c` as the external downloader for DASH(mpd) and HLS(m3u8) formats
 
-* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula, ukcolumn, whowatch, MxplayerShow, parlview (au), YoutubeWebArchive, fancode, Saitosan, ShemarooMe, telemundo, VootSeries, SonyLIVSeries, HotstarSeries, VidioPremier, VidioLive
+* **New extractors**: AnimeLab, Philo MSO, Spectrum MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula, ukcolumn, whowatch, MxplayerShow, parlview (au), YoutubeWebArchive, fancode, Saitosan, ShemarooMe, telemundo, VootSeries, SonyLIVSeries, HotstarSeries, VidioPremier, VidioLive, RCTIPlus, TBS Live
 
-* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, akamai, ina, rumble, tennistv, amcnetworks, la7 podcasts, linuxacadamy, nitter, twitcasting, viu, crackle, curiositystream, mediasite, rmcdecouverte, sonyliv, tubi, tenplay, patreon
+* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, akamai, ina, rumble, tennistv, amcnetworks, la7 podcasts, linuxacadamy, nitter, twitcasting, viu, crackle, curiositystream, mediasite, rmcdecouverte, sonyliv, tubi, tenplay, patreon, videa, yahoo
 
 * **Subtitle extraction from manifests**: Subtitles can be extracted from streaming media manifests. See [commit/be6202f](https://github.com/yt-dlp/yt-dlp/commit/be6202f12b97858b9d716e608394b51065d0419f) for details
 
@@ -127,6 +128,7 @@ Some of yt-dlp's default options are different from that of youtube-dl and youtu
 * `--add-metadata` attaches the `infojson` to `mkv` files in addition to writing the metadata when used with `--write-infojson`. Use `--compat-options no-attach-info-json` to revert this
 * `playlist_index` behaves differently when used with options like `--playlist-reverse` and `--playlist-items`. See [#302](https://github.com/yt-dlp/yt-dlp/issues/302) for details. You can use `--compat-options playlist-index` if you want to keep the earlier behavior
 * The output of `-F` is listed in a new format. Use `--compat-options list-formats` to revert this
+* All *experiences* of a funimation episode are considered as a single video. This behavior breaks existing archives. Use `--compat-options seperate-video-versions` to extract information from only the default player
 * Youtube live chat (if available) is considered as a subtitle. Use `--sub-langs all,-live_chat` to download all subtitles except live chat. You can also use `--compat-options no-live-chat` to prevent live chat from downloading
 * Youtube channel URLs are automatically redirected to `/video`. Append a `/featured` to the URL to download only the videos in the home page. If the channel does not have a videos tab, we try to download the equivalent `UU` playlist instead. Also, `/live` URLs raise an error if there are no live videos instead of silently downloading the entire channel. You may use `--compat-options no-youtube-channel-redirect` to revert all these redirections
 * Unavailable videos are also listed for youtube playlists. Use `--compat-options no-youtube-unavailable-videos` to remove this
@@ -433,7 +435,8 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --downloader-args NAME:ARGS      Give these arguments to the external
                                      downloader. Specify the downloader name and
                                      the arguments separated by a colon ":". You
-                                     can use this option multiple times
+                                     can use this option multiple times to give
+                                     different arguments to different downloaders
                                      (Alias: --external-downloader-args)
 
 ## Filesystem Options:
@@ -816,18 +819,10 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --no-hls-split-discontinuity     Do not split HLS playlists to different
                                      formats at discontinuities such as ad
                                      breaks (default)
-    --youtube-include-dash-manifest  Download the DASH manifests and related
+    --extractor-args KEY:ARGS        Pass these arguments to the extractor. See
-                                     data on YouTube videos (default)
+                                     "EXTRACTOR ARGUMENTS" for details. You can
-                                     (Alias: --no-youtube-skip-dash-manifest)
+                                     use this option multiple times to give
-    --youtube-skip-dash-manifest     Do not download the DASH manifests and
+                                     different arguments to different extractors
-                                     related data on YouTube videos
-                                     (Alias: --no-youtube-include-dash-manifest)
-    --youtube-include-hls-manifest   Download the HLS manifests and related data
-                                     on YouTube videos (default)
-                                     (Alias: --no-youtube-skip-hls-manifest)
-    --youtube-skip-hls-manifest      Do not download the HLS manifests and
-                                     related data on YouTube videos
-                                     (Alias: --no-youtube-include-hls-manifest)
 
 # CONFIGURATION
 
@@ -1021,7 +1016,7 @@ Available only when used in `--print`:
 
 Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with placeholder value provided with `--output-na-placeholder` (`NA` by default).
 
-For example for `-o %(title)s-%(id)s.%(ext)s` and an mp4 video with title `yt-dlp test video` and id `BaW_jenozKcj`, this will result in a `yt-dlp test video-BaW_jenozKcj.mp4` file created in the current directory.
+For example for `-o %(title)s-%(id)s.%(ext)s` and an mp4 video with title `yt-dlp test video` and id `BaW_jenozKc`, this will result in a `yt-dlp test video-BaW_jenozKc.mp4` file created in the current directory.
 
 For numeric sequences you can use numeric related formatting, for example, `%(view_count)05d` will result in a string with view count padded with zeros up to 5 characters, like in `00042`.
 
@@ -1331,6 +1326,23 @@ $ yt-dlp --parse-metadata 'description:(?s)(?P<meta_comment>.+)' --add-metadata
 
 ```
 
+# EXTRACTOR ARGUMENTS
+
+Some extractors accept additional arguments which can be passed using `--extractor-args KEY:ARGS`. `ARGS` is a `;` (semicolon) seperated string of `ARG=VAL1,VAL2`. Eg: `--extractor-args "youtube:skip=dash,hls;player_client=android" --extractor-args "funimation:version=uncut"`
+
+The following extractors use this feature:
+* **youtube**
+    * `skip`: `hls` or `dash` (or both) to skip download of the respective manifests
+    * `player_client`: `web` (default) or `android` (force use the android client fallbacks for video extraction)
+    * `player_skip`: `configs` - skip requests if applicable for client configs and use defaults
+
+* **funimation**
+    * `language`: Languages to extract. Eg: `funimation:language=english,japanese`
+    * `version`: The video version to extract - `uncut` or `simulcast`
+
+NOTE: These options may be changed/removed in the future without concern for backward compatibility
+
+
 # PLUGINS
 
 Plugins are loaded from `<root-dir>/ytdlp_plugins/<type>/__init__.py`. Currently only `extractor` plugins are supported. Support for `downloader` and `postprocessor` plugins may be added in the future. See [ytdlp_plugins](ytdlp_plugins) for example.
@@ -1362,6 +1374,10 @@ While these options still work, their use is not recommended since there are oth
     --list-formats-old               --compat-options list-formats (Alias: --no-list-formats-as-table)
     --list-formats-as-table          --compat-options -list-formats [Default] (Alias: --no-list-formats-old)
     --sponskrub-args ARGS            --ppa "sponskrub:ARGS"
+    --youtube-skip-dash-manifest     --extractor-args "youtube:skip=dash" (Alias: --no-youtube-include-dash-manifest)
+    --youtube-skip-hls-manifest      --extractor-args "youtube:skip=hls" (Alias: --no-youtube-include-hls-manifest)
+    --youtube-include-dash-manifest  Default (Alias: --no-youtube-skip-dash-manifest)
+    --youtube-include-hls-manifest   Default (Alias: --no-youtube-skip-hls-manifest)
     --test                           Used by developers for testing extractors. Not intended for the end user
     --youtube-print-sig-code         Used for testing youtube signatures
 

test/test_utils.py

@@ -1545,8 +1545,8 @@ Line 1
         self.assertEqual(repr(LazyList(it)), repr(it))
         self.assertEqual(str(LazyList(it)), str(it))
 
-        self.assertEqual(list(reversed(LazyList(it))), it[::-1])
+        self.assertEqual(list(LazyList(it).reverse()), it[::-1])
-        self.assertEqual(list(reversed(LazyList(it))[1:3:7]), it[::-1][1:3:7])
+        self.assertEqual(list(LazyList(it).reverse()[1:3:7]), it[::-1][1:3:7])
 
     def test_LazyList_laziness(self):
 
@@ -1559,13 +1559,13 @@ Line 1
         test(ll, 5, 5, range(6))
         test(ll, -3, 7, range(10))
 
-        ll = reversed(LazyList(range(10)))
+        ll = LazyList(range(10)).reverse()
         test(ll, -1, 0, range(1))
         test(ll, 3, 6, range(10))
 
         ll = LazyList(itertools.count())
         test(ll, 10, 10, range(11))
-        reversed(ll)
+        ll.reverse()
         test(ll, -15, 14, range(15))
 
 

yt_dlp/YoutubeDL.py

@@ -104,6 +104,7 @@ from .utils import (
     ThrottledDownload,
     to_high_limit_path,
     traverse_obj,
+    try_get,
     UnavailableVideoError,
     url_basename,
     version_tuple,
@@ -391,11 +392,9 @@ class YoutubeDL(object):
                        if True, otherwise use ffmpeg/avconv if False, otherwise
                        use downloader suggested by extractor if None.
     compat_opts:       Compatibility options. See "Differences in default behavior".
-                       Note that only format-sort, format-spec, no-live-chat,
+                       The following options do not work when used through the API:
-                       no-attach-info-json, playlist-index, list-formats,
+                       filename, abort-on-error, multistreams, no-live-chat,
-                       no-direct-merge, embed-thumbnail-atomicparsley,
+                       no-playlist-metafiles. Refer __init__.py for their implementation
-                       no-youtube-unavailable-videos, no-youtube-channel-redirect,
-                       works when used via the API
 
     The following parameters are not used by YoutubeDL itself, they are used by
     the downloader (see yt_dlp/downloader/common.py):
@@ -419,11 +418,16 @@ class YoutubeDL(object):
     dynamic_mpd:       Whether to process dynamic DASH manifests (default: True)
     hls_split_discontinuity: Split HLS playlists to different formats at
                        discontinuities such as ad breaks (default: False)
-    youtube_include_dash_manifest: If True (default), DASH manifests and related
+    extractor_args:    A dictionary of arguments to be passed to the extractors.
+                       See "EXTRACTOR ARGUMENTS" for details.
+                       Eg: {'youtube': {'skip': ['dash', 'hls']}}
+    youtube_include_dash_manifest: Deprecated - Use extractor_args instead.
+                       If True (default), DASH manifests and related
                        data will be downloaded and processed by extractor.
                        You can reduce network I/O by disabling it if you don't
                        care about DASH. (only for youtube)
-    youtube_include_hls_manifest: If True (default), HLS manifests and related
+    youtube_include_hls_manifest: Deprecated - Use extractor_args instead.
+                       If True (default), HLS manifests and related
                        data will be downloaded and processed by extractor.
                        You can reduce network I/O by disabling it if you don't
                        care about HLS. (only for youtube)
@@ -1176,13 +1180,17 @@ class YoutubeDL(object):
             return ie_result
 
     def add_default_extra_info(self, ie_result, ie, url):
-        self.add_extra_info(ie_result, {
+        if url is not None:
-            'extractor': ie.IE_NAME,
+            self.add_extra_info(ie_result, {
-            'webpage_url': url,
+                'webpage_url': url,
-            'original_url': url,
+                'original_url': url,
-            'webpage_url_basename': url_basename(url),
+                'webpage_url_basename': url_basename(url),
-            'extractor_key': ie.ie_key(),
+            })
-        })
+        if ie is not None:
+            self.add_extra_info(ie_result, {
+                'extractor': ie.IE_NAME,
+                'extractor_key': ie.ie_key(),
+            })
 
     def process_ie_result(self, ie_result, download=True, extra_info={}):
         """
@@ -1201,8 +1209,8 @@ class YoutubeDL(object):
                     or extract_flat is True):
                 info_copy = ie_result.copy()
                 self.add_extra_info(info_copy, extra_info)
-                self.add_default_extra_info(
+                ie = try_get(ie_result.get('ie_key'), self.get_info_extractor)
-                    info_copy, self.get_info_extractor(ie_result.get('ie_key')), ie_result['url'])
+                self.add_default_extra_info(info_copy, ie, ie_result['url'])
                 self.__forced_printings(info_copy, self.prepare_filename(info_copy), incomplete=True)
                 return ie_result
 
@@ -1748,6 +1756,8 @@ class YoutubeDL(object):
             return new_dict
 
         def _check_formats(formats):
+            if not check_formats:
+                yield from formats
             for f in formats:
                 self.to_screen('[info] Testing format %s' % f['format_id'])
                 temp_file = tempfile.NamedTemporaryFile(
@@ -1755,16 +1765,16 @@ class YoutubeDL(object):
                     dir=self.get_output_path('temp') or None)
                 temp_file.close()
                 try:
-                    dl, _ = self.dl(temp_file.name, f, test=True)
+                    success, _ = self.dl(temp_file.name, f, test=True)
-                except (ExtractorError, IOError, OSError, ValueError) + network_exceptions:
+                except (DownloadError, IOError, OSError, ValueError) + network_exceptions:
-                    dl = False
+                    success = False
                 finally:
                     if os.path.exists(temp_file.name):
                         try:
                             os.remove(temp_file.name)
                         except OSError:
                             self.report_warning('Unable to delete temporary file "%s"' % temp_file.name)
-                if dl:
+                if success:
                     yield f
                 else:
                     self.to_screen('[info] Unable to download format %s. Skipping...' % f['format_id'])
@@ -1775,8 +1785,7 @@ class YoutubeDL(object):
 
                 def selector_function(ctx):
                     for f in fs:
-                        for format in f(ctx):
+                        yield from f(ctx)
-                            yield format
                 return selector_function
 
             elif selector.type == GROUP:  # ()
@@ -1792,22 +1801,24 @@ class YoutubeDL(object):
                             return picked_formats
                     return []
 
+            elif selector.type == MERGE:  # +
+                selector_1, selector_2 = map(_build_selector_function, selector.selector)
+
+                def selector_function(ctx):
+                    for pair in itertools.product(
+                            selector_1(copy.deepcopy(ctx)), selector_2(copy.deepcopy(ctx))):
+                        yield _merge(pair)
+
             elif selector.type == SINGLE:  # atom
                 format_spec = selector.selector or 'best'
 
                 # TODO: Add allvideo, allaudio etc by generalizing the code with best/worst selector
                 if format_spec == 'all':
                     def selector_function(ctx):
-                        formats = list(ctx['formats'])
+                        yield from _check_formats(ctx['formats'])
-                        if check_formats:
-                            formats = _check_formats(formats)
-                        for f in formats:
-                            yield f
                 elif format_spec == 'mergeall':
                     def selector_function(ctx):
-                        formats = ctx['formats']
+                        formats = list(_check_formats(ctx['formats']))
-                        if check_formats:
-                            formats = list(_check_formats(formats))
                         if not formats:
                             return
                         merged_format = formats[-1]
@@ -1845,29 +1856,17 @@ class YoutubeDL(object):
 
                     def selector_function(ctx):
                         formats = list(ctx['formats'])
-                        if not formats:
-                            return
                         matches = list(filter(filter_f, formats)) if filter_f is not None else formats
                         if format_fallback and ctx['incomplete_formats'] and not matches:
                             # for extractors with incomplete formats (audio only (soundcloud)
                             # or video only (imgur)) best/worst will fallback to
                             # best/worst {video,audio}-only format
                             matches = formats
-                        if format_reverse:
+                        matches = LazyList(_check_formats(matches[::-1 if format_reverse else 1]))
-                            matches = matches[::-1]
+                        try:
-                        if check_formats:
-                            matches = list(itertools.islice(_check_formats(matches), format_idx))
-                        n = len(matches)
-                        if -n <= format_idx - 1 < n:
                             yield matches[format_idx - 1]
-
+                        except IndexError:
-            elif selector.type == MERGE:        # +
+                            return
-                selector_1, selector_2 = map(_build_selector_function, selector.selector)
-
-                def selector_function(ctx):
-                    for pair in itertools.product(
-                            selector_1(copy.deepcopy(ctx)), selector_2(copy.deepcopy(ctx))):
-                        yield _merge(pair)
 
             filters = [self._build_format_filter(f) for f in selector.filters]
 
@@ -1961,7 +1960,7 @@ class YoutubeDL(object):
                     t['resolution'] = '%dx%d' % (t['width'], t['height'])
                 t['url'] = sanitize_url(t['url'])
             if self.params.get('check_formats'):
-                info_dict['thumbnails'] = reversed(LazyList(filter(test_thumbnail, thumbnails[::-1])))
+                info_dict['thumbnails'] = LazyList(filter(test_thumbnail, thumbnails[::-1])).reverse()
 
     def process_video_result(self, info_dict, download=True):
         assert info_dict.get('_type', 'video') == 'video'
@@ -2001,10 +2000,6 @@ class YoutubeDL(object):
 
         self._sanitize_thumbnails(info_dict)
 
-        if self.params.get('list_thumbnails'):
-            self.list_thumbnails(info_dict)
-            return
-
         thumbnail = info_dict.get('thumbnail')
         thumbnails = info_dict.get('thumbnails')
         if thumbnail:
@@ -2047,13 +2042,6 @@ class YoutubeDL(object):
         automatic_captions = info_dict.get('automatic_captions')
         subtitles = info_dict.get('subtitles')
 
-        if self.params.get('listsubtitles', False):
-            if 'automatic_captions' in info_dict:
-                self.list_subtitles(
-                    info_dict['id'], automatic_captions, 'automatic captions')
-            self.list_subtitles(info_dict['id'], subtitles, 'subtitles')
-            return
-
         info_dict['requested_subtitles'] = self.process_subtitles(
             info_dict['id'], subtitles, automatic_captions)
 
@@ -2141,10 +2129,20 @@ class YoutubeDL(object):
 
         info_dict, _ = self.pre_process(info_dict)
 
-        if self.params.get('listformats'):
+        list_only = self.params.get('list_thumbnails') or self.params.get('listformats') or self.params.get('listsubtitles')
-            if not info_dict.get('formats'):
+        if list_only:
-                raise ExtractorError('No video formats found', expected=True)
+            self.__forced_printings(info_dict, self.prepare_filename(info_dict), incomplete=True)
-            self.list_formats(info_dict)
+            if self.params.get('list_thumbnails'):
+                self.list_thumbnails(info_dict)
+            if self.params.get('listformats'):
+                if not info_dict.get('formats'):
+                    raise ExtractorError('No video formats found', expected=True)
+                self.list_formats(info_dict)
+            if self.params.get('listsubtitles'):
+                if 'automatic_captions' in info_dict:
+                    self.list_subtitles(
+                        info_dict['id'], automatic_captions, 'automatic captions')
+                self.list_subtitles(info_dict['id'], subtitles, 'subtitles')
             return
 
         format_selector = self.format_selector
@@ -2185,6 +2183,8 @@ class YoutubeDL(object):
                 raise ExtractorError('Requested format is not available', expected=True)
             else:
                 self.report_warning('Requested format is not available')
+                # Process what we can, even without any available formats.
+                self.process_info(dict(info_dict))
         elif download:
             self.to_screen(
                 '[info] %s: Downloading %d format(s): %s' % (
@@ -2349,7 +2349,7 @@ class YoutubeDL(object):
         # TODO: backward compatibility, to be removed
         info_dict['fulltitle'] = info_dict['title']
 
-        if 'format' not in info_dict:
+        if 'format' not in info_dict and 'ext' in info_dict:
             info_dict['format'] = info_dict['ext']
 
         if self._match_entry(info_dict) is not None:
@@ -2364,7 +2364,7 @@ class YoutubeDL(object):
         files_to_move = {}
 
         # Forced printings
-        self.__forced_printings(info_dict, full_filename, incomplete=False)
+        self.__forced_printings(info_dict, full_filename, incomplete=('format' not in info_dict))
 
         if self.params.get('simulate', False):
             if self.params.get('force_write_download_archive', False):
@@ -2791,7 +2791,7 @@ class YoutubeDL(object):
             info = self.filter_requested_info(json.loads('\n'.join(f)), self.params.get('clean_infojson', True))
         try:
             self.process_ie_result(info, download=True)
-        except (DownloadError, EntryNotInPlaylist):
+        except (DownloadError, EntryNotInPlaylist, ThrottledDownload):
             webpage_url = info.get('webpage_url')
             if webpage_url is not None:
                 self.report_warning('The info failed to download, trying with "%s"' % webpage_url)
@@ -3010,7 +3010,7 @@ class YoutubeDL(object):
         formats = info_dict.get('formats', [info_dict])
         new_format = (
             'list-formats' not in self.params.get('compat_opts', [])
-            and self.params.get('list_formats_as_table', True) is not False)
+            and self.params.get('listformats_table', True) is not False)
         if new_format:
             table = [
                 [
@@ -3045,12 +3045,9 @@ class YoutubeDL(object):
             header_line = ['format code', 'extension', 'resolution', 'note']
 
         self.to_screen(
-            '[info] Available formats for %s:\n%s' % (info_dict['id'], render_table(
+            '[info] Available formats for %s:' % info_dict['id'])
-                header_line,
+        self.to_stdout(render_table(
-                table,
+            header_line, table, delim=new_format, extraGap=(0 if new_format else 1), hideEmpty=new_format))
-                delim=new_format,
-                extraGap=(0 if new_format else 1),
-                hideEmpty=new_format)))
 
     def list_thumbnails(self, info_dict):
         thumbnails = list(info_dict.get('thumbnails'))
@@ -3060,7 +3057,7 @@ class YoutubeDL(object):
 
         self.to_screen(
             '[info] Thumbnails for %s:' % info_dict['id'])
-        self.to_screen(render_table(
+        self.to_stdout(render_table(
             ['ID', 'width', 'height', 'URL'],
             [[t['id'], t.get('width', 'unknown'), t.get('height', 'unknown'), t['url']] for t in thumbnails]))
 
@@ -3072,12 +3069,12 @@ class YoutubeDL(object):
             'Available %s for %s:' % (name, video_id))
 
         def _row(lang, formats):
-            exts, names = zip(*((f['ext'], f.get('name', 'unknown')) for f in reversed(formats)))
+            exts, names = zip(*((f['ext'], f.get('name') or 'unknown') for f in reversed(formats)))
             if len(set(names)) == 1:
                 names = [] if names[0] == 'unknown' else names[:1]
             return [lang, ', '.join(names), ', '.join(exts)]
 
-        self.to_screen(render_table(
+        self.to_stdout(render_table(
             ['Language', 'Name', 'Formats'],
             [_row(lang, formats) for lang, formats in subtitles.items()],
             hideEmpty=True))
@@ -3255,7 +3252,7 @@ class YoutubeDL(object):
         multiple = write_all and len(thumbnails) > 1
 
         ret = []
-        for t in thumbnails[::1 if write_all else -1]:
+        for t in thumbnails[::-1]:
             thumb_ext = determine_ext(t['url'], 'jpg')
             suffix = '%s.' % t['id'] if multiple else ''
             thumb_display_id = '%s ' % t['id'] if multiple else ''

yt_dlp/__init__.py

@@ -273,7 +273,7 @@ def _real_main(argv=None):
         'filename', 'format-sort', 'abort-on-error', 'format-spec', 'no-playlist-metafiles',
         'multistreams', 'no-live-chat', 'playlist-index', 'list-formats', 'no-direct-merge',
         'no-youtube-channel-redirect', 'no-youtube-unavailable-videos', 'no-attach-info-json',
-        'embed-thumbnail-atomicparsley',
+        'embed-thumbnail-atomicparsley', 'seperate-video-versions',
     ]
     compat_opts = parse_compat_opts()
 
@@ -631,6 +631,7 @@ def _real_main(argv=None):
         'include_ads': opts.include_ads,
         'default_search': opts.default_search,
         'dynamic_mpd': opts.dynamic_mpd,
+        'extractor_args': opts.extractor_args,
         'youtube_include_dash_manifest': opts.youtube_include_dash_manifest,
         'youtube_include_hls_manifest': opts.youtube_include_hls_manifest,
         'encoding': opts.encoding,

yt_dlp/downloader/dash.py

@@ -57,9 +57,6 @@ class DashSegmentsFD(FragmentFD):
             # TODO: Make progress updates work without hooking twice
             # for ph in self._progress_hooks:
             #     fd.add_progress_hook(ph)
-            success = fd.real_download(filename, info_copy)
+            return fd.real_download(filename, info_copy)
-            if not success:
+
-                return False
+        return self.download_and_append_fragments(ctx, fragments_to_download, info_dict)
-        else:
-            self.download_and_append_fragments(ctx, fragments_to_download, info_dict)
-        return True

yt_dlp/downloader/external.py

@@ -377,8 +377,6 @@ class FFmpegFD(ExternalFD):
             # http://trac.ffmpeg.org/ticket/6125#comment:10
             args += ['-seekable', '1' if seekable else '0']
 
-        args += self._configuration_args()
-
         # start_time = info_dict.get('start_time') or 0
         # if start_time:
         #     args += ['-ss', compat_str(start_time)]
@@ -446,7 +444,8 @@ class FFmpegFD(ExternalFD):
 
         for url in urls:
             args += ['-i', url]
-        args += ['-c', 'copy']
+
+        args += self._configuration_args() + ['-c', 'copy']
         if info_dict.get('requested_formats'):
             for (i, fmt) in enumerate(info_dict['requested_formats']):
                 if fmt.get('acodec') != 'none':

yt_dlp/downloader/fragment.py

@@ -328,8 +328,7 @@ class FragmentFD(FileDownloader):
 
     def download_and_append_fragments(self, ctx, fragments, info_dict, pack_func=None):
         fragment_retries = self.params.get('fragment_retries', 0)
-        skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
+        is_fatal = (lambda idx: idx == 0) if self.params.get('skip_unavailable_fragments', True) else (lambda _: True)
-        test = self.params.get('test', False)
         if not pack_func:
             pack_func = lambda frag_content, _: frag_content
 
@@ -341,7 +340,7 @@ class FragmentFD(FileDownloader):
                 headers['Range'] = 'bytes=%d-%d' % (byte_range['start'], byte_range['end'] - 1)
 
             # Never skip the first fragment
-            fatal = (fragment.get('index') or frag_index) == 0 or not skip_unavailable_fragments
+            fatal = is_fatal(fragment.get('index') or (frag_index - 1))
             count, frag_content = 0, None
             while count <= fragment_retries:
                 try:
@@ -382,14 +381,13 @@ class FragmentFD(FileDownloader):
             # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
             # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
             # not what it decrypts to.
-            if test:
+            if self.params.get('test', False):
                 return frag_content
             return AES.new(decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
 
         def append_fragment(frag_content, frag_index, ctx):
             if not frag_content:
-                fatal = frag_index == 1 or not skip_unavailable_fragments
+                if not is_fatal(frag_index - 1):
-                if not fatal:
                     self.report_skip_fragment(frag_index)
                     return True
                 else:
@@ -404,13 +402,9 @@ class FragmentFD(FileDownloader):
         if can_threaded_download and max_workers > 1:
 
             def _download_fragment(fragment):
-                try:
+                ctx_copy = ctx.copy()
-                    ctx_copy = ctx.copy()
+                frag_content, frag_index = download_fragment(fragment, ctx_copy)
-                    frag_content, frag_index = download_fragment(fragment, ctx_copy)
+                return fragment, frag_content, frag_index, ctx_copy.get('fragment_filename_sanitized')
-                    return fragment, frag_content, frag_index, ctx_copy.get('fragment_filename_sanitized')
-                except Exception:
-                    # Return immediately on exception so that it is raised in the main thread
-                    return
 
             self.report_warning('The download speed shown is only of one thread. This is a known issue and patches are welcome')
             with concurrent.futures.ThreadPoolExecutor(max_workers) as pool:
@@ -428,3 +422,4 @@ class FragmentFD(FileDownloader):
                     return False
 
         self._finish_frag_download(ctx)
+        return True

yt_dlp/downloader/hls.py

@@ -250,78 +250,75 @@ class HlsFD(FragmentFD):
             # TODO: Make progress updates work without hooking twice
             # for ph in self._progress_hooks:
             #     fd.add_progress_hook(ph)
-            success = fd.real_download(filename, info_copy)
+            return fd.real_download(filename, info_copy)
-            if not success:
+
-                return False
+        if is_webvtt:
+            def pack_fragment(frag_content, frag_index):
+                output = io.StringIO()
+                adjust = 0
+                for block in webvtt.parse_fragment(frag_content):
+                    if isinstance(block, webvtt.CueBlock):
+                        block.start += adjust
+                        block.end += adjust
+
+                        dedup_window = extra_state.setdefault('webvtt_dedup_window', [])
+                        cue = block.as_json
+
+                        # skip the cue if an identical one appears
+                        # in the window of potential duplicates
+                        # and prune the window of unviable candidates
+                        i = 0
+                        skip = True
+                        while i < len(dedup_window):
+                            window_cue = dedup_window[i]
+                            if window_cue == cue:
+                                break
+                            if window_cue['end'] >= cue['start']:
+                                i += 1
+                                continue
+                            del dedup_window[i]
+                        else:
+                            skip = False
+
+                        if skip:
+                            continue
+
+                        # add the cue to the window
+                        dedup_window.append(cue)
+                    elif isinstance(block, webvtt.Magic):
+                        # take care of MPEG PES timestamp overflow
+                        if block.mpegts is None:
+                            block.mpegts = 0
+                        extra_state.setdefault('webvtt_mpegts_adjust', 0)
+                        block.mpegts += extra_state['webvtt_mpegts_adjust'] << 33
+                        if block.mpegts < extra_state.get('webvtt_mpegts_last', 0):
+                            extra_state['webvtt_mpegts_adjust'] += 1
+                            block.mpegts += 1 << 33
+                        extra_state['webvtt_mpegts_last'] = block.mpegts
+
+                        if frag_index == 1:
+                            extra_state['webvtt_mpegts'] = block.mpegts or 0
+                            extra_state['webvtt_local'] = block.local or 0
+                            # XXX: block.local = block.mpegts = None ?
+                        else:
+                            if block.mpegts is not None and block.local is not None:
+                                adjust = (
+                                    (block.mpegts - extra_state.get('webvtt_mpegts', 0))
+                                    - (block.local - extra_state.get('webvtt_local', 0))
+                                )
+                            continue
+                    elif isinstance(block, webvtt.HeaderBlock):
+                        if frag_index != 1:
+                            # XXX: this should probably be silent as well
+                            # or verify that all segments contain the same data
+                            self.report_warning(bug_reports_message(
+                                'Discarding a %s block found in the middle of the stream; '
+                                'if the subtitles display incorrectly,'
+                                % (type(block).__name__)))
+                            continue
+                    block.write_into(output)
+
+                return output.getvalue().encode('utf-8')
         else:
-            if is_webvtt:
+            pack_fragment = None
-                def pack_fragment(frag_content, frag_index):
+        return self.download_and_append_fragments(ctx, fragments, info_dict, pack_fragment)
-                    output = io.StringIO()
-                    adjust = 0
-                    for block in webvtt.parse_fragment(frag_content):
-                        if isinstance(block, webvtt.CueBlock):
-                            block.start += adjust
-                            block.end += adjust
-
-                            dedup_window = extra_state.setdefault('webvtt_dedup_window', [])
-                            cue = block.as_json
-
-                            # skip the cue if an identical one appears
-                            # in the window of potential duplicates
-                            # and prune the window of unviable candidates
-                            i = 0
-                            skip = True
-                            while i < len(dedup_window):
-                                window_cue = dedup_window[i]
-                                if window_cue == cue:
-                                    break
-                                if window_cue['end'] >= cue['start']:
-                                    i += 1
-                                    continue
-                                del dedup_window[i]
-                            else:
-                                skip = False
-
-                            if skip:
-                                continue
-
-                            # add the cue to the window
-                            dedup_window.append(cue)
-                        elif isinstance(block, webvtt.Magic):
-                            # take care of MPEG PES timestamp overflow
-                            if block.mpegts is None:
-                                block.mpegts = 0
-                            extra_state.setdefault('webvtt_mpegts_adjust', 0)
-                            block.mpegts += extra_state['webvtt_mpegts_adjust'] << 33
-                            if block.mpegts < extra_state.get('webvtt_mpegts_last', 0):
-                                extra_state['webvtt_mpegts_adjust'] += 1
-                                block.mpegts += 1 << 33
-                            extra_state['webvtt_mpegts_last'] = block.mpegts
-
-                            if frag_index == 1:
-                                extra_state['webvtt_mpegts'] = block.mpegts or 0
-                                extra_state['webvtt_local'] = block.local or 0
-                                # XXX: block.local = block.mpegts = None ?
-                            else:
-                                if block.mpegts is not None and block.local is not None:
-                                    adjust = (
-                                        (block.mpegts - extra_state.get('webvtt_mpegts', 0))
-                                        - (block.local - extra_state.get('webvtt_local', 0))
-                                    )
-                                continue
-                        elif isinstance(block, webvtt.HeaderBlock):
-                            if frag_index != 1:
-                                # XXX: this should probably be silent as well
-                                # or verify that all segments contain the same data
-                                self.report_warning(bug_reports_message(
-                                    'Discarding a %s block found in the middle of the stream; '
-                                    'if the subtitles display incorrectly,'
-                                    % (type(block).__name__)))
-                                continue
-                        block.write_into(output)
-
-                    return output.getvalue().encode('utf-8')
-            else:
-                pack_fragment = None
-            self.download_and_append_fragments(ctx, fragments, info_dict, pack_fragment)
-        return True

yt_dlp/downloader/youtube_live_chat.py

@@ -44,7 +44,7 @@ class YoutubeLiveChatFD(FragmentFD):
             return self._download_fragment(ctx, url, info_dict, http_headers, data)
 
         def parse_actions_replay(live_chat_continuation):
-            offset = continuation_id = None
+            offset = continuation_id = click_tracking_params = None
             processed_fragment = bytearray()
             for action in live_chat_continuation.get('actions', []):
                 if 'replayChatItemAction' in action:
@@ -53,17 +53,34 @@ class YoutubeLiveChatFD(FragmentFD):
                 processed_fragment.extend(
                     json.dumps(action, ensure_ascii=False).encode('utf-8') + b'\n')
             if offset is not None:
-                continuation_id = try_get(
+                continuation = try_get(
                     live_chat_continuation,
-                    lambda x: x['continuations'][0]['liveChatReplayContinuationData']['continuation'])
+                    lambda x: x['continuations'][0]['liveChatReplayContinuationData'], dict)
+                if continuation:
+                    continuation_id = continuation.get('continuation')
+                    click_tracking_params = continuation.get('clickTrackingParams')
             self._append_fragment(ctx, processed_fragment)
-            return continuation_id, offset
+            return continuation_id, offset, click_tracking_params
+
+        def try_refresh_replay_beginning(live_chat_continuation):
+            # choose the second option that contains the unfiltered live chat replay
+            refresh_continuation = try_get(
+                live_chat_continuation,
+                lambda x: x['header']['liveChatHeaderRenderer']['viewSelector']['sortFilterSubMenuRenderer']['subMenuItems'][1]['continuation']['reloadContinuationData'], dict)
+            if refresh_continuation:
+                # no data yet but required to call _append_fragment
+                self._append_fragment(ctx, b'')
+                refresh_continuation_id = refresh_continuation.get('continuation')
+                offset = 0
+                click_tracking_params = refresh_continuation.get('trackingParams')
+                return refresh_continuation_id, offset, click_tracking_params
+            return parse_actions_replay(live_chat_continuation)
 
         live_offset = 0
 
         def parse_actions_live(live_chat_continuation):
             nonlocal live_offset
-            continuation_id = None
+            continuation_id = click_tracking_params = None
             processed_fragment = bytearray()
             for action in live_chat_continuation.get('actions', []):
                 timestamp = self.parse_live_timestamp(action)
@@ -84,37 +101,44 @@ class YoutubeLiveChatFD(FragmentFD):
             continuation_data = try_get(live_chat_continuation, continuation_data_getters, dict)
             if continuation_data:
                 continuation_id = continuation_data.get('continuation')
+                click_tracking_params = continuation_data.get('clickTrackingParams')
                 timeout_ms = int_or_none(continuation_data.get('timeoutMs'))
                 if timeout_ms is not None:
                     time.sleep(timeout_ms / 1000)
             self._append_fragment(ctx, processed_fragment)
-            return continuation_id, live_offset
+            return continuation_id, live_offset, click_tracking_params
 
-        if info_dict['protocol'] == 'youtube_live_chat_replay':
+        def download_and_parse_fragment(url, frag_index, request_data=None, headers=None):
-            parse_actions = parse_actions_replay
-        elif info_dict['protocol'] == 'youtube_live_chat':
-            parse_actions = parse_actions_live
-
-        def download_and_parse_fragment(url, frag_index, request_data, headers):
             count = 0
             while count <= fragment_retries:
                 try:
                     success, raw_fragment = dl_fragment(url, request_data, headers)
                     if not success:
-                        return False, None, None
+                        return False, None, None, None
-                    data = json.loads(raw_fragment)
+                    try:
+                        data = ie._extract_yt_initial_data(video_id, raw_fragment.decode('utf-8', 'replace'))
+                    except RegexNotFoundError:
+                        data = None
+                    if not data:
+                        data = json.loads(raw_fragment)
                     live_chat_continuation = try_get(
                         data,
                         lambda x: x['continuationContents']['liveChatContinuation'], dict) or {}
-                    continuation_id, offset = parse_actions(live_chat_continuation)
+                    if info_dict['protocol'] == 'youtube_live_chat_replay':
-                    return True, continuation_id, offset
+                        if frag_index == 1:
+                            continuation_id, offset, click_tracking_params = try_refresh_replay_beginning(live_chat_continuation)
+                        else:
+                            continuation_id, offset, click_tracking_params = parse_actions_replay(live_chat_continuation)
+                    elif info_dict['protocol'] == 'youtube_live_chat':
+                        continuation_id, offset, click_tracking_params = parse_actions_live(live_chat_continuation)
+                    return True, continuation_id, offset, click_tracking_params
                 except compat_urllib_error.HTTPError as err:
                     count += 1
                     if count <= fragment_retries:
                         self.report_retry_fragment(err, frag_index, count, fragment_retries)
             if count > fragment_retries:
                 self.report_error('giving up after %s fragment retries' % fragment_retries)
-                return False, None, None
+                return False, None, None, None
 
         self._prepare_and_start_frag_download(ctx)
 
@@ -142,10 +166,13 @@ class YoutubeLiveChatFD(FragmentFD):
         visitor_data = try_get(innertube_context, lambda x: x['client']['visitorData'], str)
         if info_dict['protocol'] == 'youtube_live_chat_replay':
             url = 'https://www.youtube.com/youtubei/v1/live_chat/get_live_chat_replay?key=' + api_key
+            chat_page_url = 'https://www.youtube.com/live_chat_replay?continuation=' + continuation_id
         elif info_dict['protocol'] == 'youtube_live_chat':
             url = 'https://www.youtube.com/youtubei/v1/live_chat/get_live_chat?key=' + api_key
+            chat_page_url = 'https://www.youtube.com/live_chat?continuation=' + continuation_id
 
         frag_index = offset = 0
+        click_tracking_params = None
         while continuation_id is not None:
             frag_index += 1
             request_data = {
@@ -154,11 +181,16 @@ class YoutubeLiveChatFD(FragmentFD):
             }
             if frag_index > 1:
                 request_data['currentPlayerState'] = {'playerOffsetMs': str(max(offset - 5000, 0))}
-            headers = ie._generate_api_headers(ytcfg, visitor_data=visitor_data)
+                if click_tracking_params:
-            headers.update({'content-type': 'application/json'})
+                    request_data['context']['clickTracking'] = {'clickTrackingParams': click_tracking_params}
-            fragment_request_data = json.dumps(request_data, ensure_ascii=False).encode('utf-8') + b'\n'
+                headers = ie._generate_api_headers(ytcfg, visitor_data=visitor_data)
-            success, continuation_id, offset = download_and_parse_fragment(
+                headers.update({'content-type': 'application/json'})
-                url, frag_index, fragment_request_data, headers)
+                fragment_request_data = json.dumps(request_data, ensure_ascii=False).encode('utf-8') + b'\n'
+                success, continuation_id, offset, click_tracking_params = download_and_parse_fragment(
+                    url, frag_index, fragment_request_data, headers)
+            else:
+                success, continuation_id, offset, click_tracking_params = download_and_parse_fragment(
+                    chat_page_url, frag_index)
             if not success:
                 return False
             if test:

yt_dlp/extractor/adobepass.py

@@ -1,6 +1,7 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import json
 import re
 import time
 import xml.etree.ElementTree as etree
@@ -61,6 +62,11 @@ MSO_INFO = {
         'username_field': 'IDToken1',
         'password_field': 'IDToken2',
     },
+    'Spectrum': {
+        'name': 'Spectrum',
+        'username_field': 'IDToken1',
+        'password_field': 'IDToken2',
+    },
     'Philo': {
         'name': 'Philo',
         'username_field': 'ident'
@@ -1524,6 +1530,41 @@ class AdobePassIE(InfoExtractor):
                         }), headers={
                             'Content-Type': 'application/x-www-form-urlencoded'
                         })
+                elif mso_id == 'Spectrum':
+                    # Spectrum's login for is dynamically loaded via JS so we need to hardcode the flow
+                    # as a one-off implementation.
+                    provider_redirect_page, urlh = provider_redirect_page_res
+                    provider_login_page_res = post_form(
+                        provider_redirect_page_res, self._DOWNLOADING_LOGIN_PAGE)
+                    saml_login_page, urlh = provider_login_page_res
+                    relay_state = self._search_regex(
+                        r'RelayState\s*=\s*"(?P<relay>.+?)";',
+                        saml_login_page, 'RelayState', group='relay')
+                    saml_request = self._search_regex(
+                        r'SAMLRequest\s*=\s*"(?P<saml_request>.+?)";',
+                        saml_login_page, 'SAMLRequest', group='saml_request')
+                    login_json = {
+                        mso_info['username_field']: username,
+                        mso_info['password_field']: password,
+                        'RelayState': relay_state,
+                        'SAMLRequest': saml_request,
+                    }
+                    saml_response_json = self._download_json(
+                        'https://tveauthn.spectrum.net/tveauthentication/api/v1/manualAuth', video_id,
+                        'Downloading SAML Response',
+                        data=json.dumps(login_json).encode(),
+                        headers={
+                            'Content-Type': 'application/json',
+                            'Accept': 'application/json',
+                        })
+                    self._download_webpage(
+                        saml_response_json['SAMLRedirectUri'], video_id,
+                        'Confirming Login', data=urlencode_postdata({
+                            'SAMLResponse': saml_response_json['SAMLResponse'],
+                            'RelayState': relay_state,
+                        }), headers={
+                            'Content-Type': 'application/x-www-form-urlencoded'
+                        })
                 else:
                     # Some providers (e.g. DIRECTV NOW) have another meta refresh
                     # based redirect that should be followed.

yt_dlp/extractor/common.py

@@ -70,6 +70,7 @@ from ..utils import (
     str_or_none,
     str_to_int,
     strip_or_none,
+    traverse_obj,
     unescapeHTML,
     unified_strdate,
     unified_timestamp,
@@ -1037,7 +1038,9 @@ class InfoExtractor(object):
             metadata_available=False, method='any'):
         if metadata_available and self.get_param('ignore_no_formats_error'):
             self.report_warning(msg)
-        raise ExtractorError('%s. %s' % (msg, self._LOGIN_HINTS[method]), expected=True)
+        if method is not None:
+            msg = '%s. %s' % (msg, self._LOGIN_HINTS[method])
+        raise ExtractorError(msg, expected=True)
 
     def raise_geo_restricted(
             self, msg='This video is not available from your location due to geo restriction',
@@ -1978,24 +1981,33 @@ class InfoExtractor(object):
             preference=None, quality=None, m3u8_id=None, live=False, note=None,
             errnote=None, fatal=True, data=None, headers={}, query={},
             video_id=None):
+        formats, subtitles = [], {}
 
         if '#EXT-X-FAXS-CM:' in m3u8_doc:  # Adobe Flash Access
-            return [], {}
+            return formats, subtitles
 
         if (not self.get_param('allow_unplayable_formats')
                 and re.search(r'#EXT-X-SESSION-KEY:.*?URI="skd://', m3u8_doc)):  # Apple FairPlay
-            return [], {}
+            return formats, subtitles
 
-        formats = []
+        def format_url(url):
+            return url if re.match(r'^https?://', url) else compat_urlparse.urljoin(m3u8_url, url)
 
-        subtitles = {}
+        if self.get_param('hls_split_discontinuity', False):
+            def _extract_m3u8_playlist_indices(manifest_url=None, m3u8_doc=None):
+                if not m3u8_doc:
+                    if not manifest_url:
+                        return []
+                    m3u8_doc = self._download_webpage(
+                        manifest_url, video_id, fatal=fatal, data=data, headers=headers,
+                        note=False, errnote='Failed to download m3u8 playlist information')
+                    if m3u8_doc is False:
+                        return []
+                return range(1 + sum(line.startswith('#EXT-X-DISCONTINUITY') for line in m3u8_doc.splitlines()))
 
-        format_url = lambda u: (
+        else:
-            u
+            def _extract_m3u8_playlist_indices(*args, **kwargs):
-            if re.match(r'^https?://', u)
+                return [None]
-            else compat_urlparse.urljoin(m3u8_url, u))
-
-        split_discontinuity = self.get_param('hls_split_discontinuity', False)
 
         # References:
         # 1. https://tools.ietf.org/html/draft-pantos-http-live-streaming-21
@@ -2013,68 +2025,16 @@ class InfoExtractor(object):
         # media playlist and MUST NOT appear in master playlist thus we can
         # clearly detect media playlist with this criterion.
 
-        def _extract_m3u8_playlist_formats(format_url=None, m3u8_doc=None, video_id=None,
-                                           fatal=True, data=None, headers={}):
-            if not m3u8_doc:
-                if not format_url:
-                    return []
-                res = self._download_webpage_handle(
-                    format_url, video_id,
-                    note=False,
-                    errnote='Failed to download m3u8 playlist information',
-                    fatal=fatal, data=data, headers=headers)
-
-                if res is False:
-                    return []
-
-                m3u8_doc, urlh = res
-                format_url = urlh.geturl()
-
-            playlist_formats = []
-            i = (
-                0
-                if split_discontinuity
-                else None)
-            format_info = {
-                'index': i,
-                'key_data': None,
-                'files': [],
-            }
-            for line in m3u8_doc.splitlines():
-                if not line.startswith('#'):
-                    format_info['files'].append(line)
-                elif split_discontinuity and line.startswith('#EXT-X-DISCONTINUITY'):
-                    i += 1
-                    playlist_formats.append(format_info)
-                    format_info = {
-                        'index': i,
-                        'url': format_url,
-                        'files': [],
-                    }
-            playlist_formats.append(format_info)
-            return playlist_formats
-
         if '#EXT-X-TARGETDURATION' in m3u8_doc:  # media playlist, return as is
-
+            formats = [{
-            playlist_formats = _extract_m3u8_playlist_formats(m3u8_doc=m3u8_doc)
+                'format_id': '-'.join(map(str, filter(None, [m3u8_id, idx]))),
-
+                'format_index': idx,
-            for format in playlist_formats:
+                'url': m3u8_url,
-                format_id = []
+                'ext': ext,
-                if m3u8_id:
+                'protocol': entry_protocol,
-                    format_id.append(m3u8_id)
+                'preference': preference,
-                format_index = format.get('index')
+                'quality': quality,
-                if format_index:
+            } for idx in _extract_m3u8_playlist_indices(m3u8_doc=m3u8_doc)]
-                    format_id.append(str(format_index))
-                f = {
-                    'format_id': '-'.join(format_id),
-                    'format_index': format_index,
-                    'url': m3u8_url,
-                    'ext': ext,
-                    'protocol': entry_protocol,
-                    'preference': preference,
-                    'quality': quality,
-                }
-                formats.append(f)
 
             return formats, subtitles
 
@@ -2114,32 +2074,19 @@ class InfoExtractor(object):
             media_url = media.get('URI')
             if media_url:
                 manifest_url = format_url(media_url)
-                format_id = []
+                formats.extend({
-                playlist_formats = _extract_m3u8_playlist_formats(manifest_url, video_id=video_id,
+                    'format_id': '-'.join(map(str, filter(None, (m3u8_id, group_id, name, idx)))),
-                                                                  fatal=fatal, data=data, headers=headers)
+                    'format_note': name,
-
+                    'format_index': idx,
-                for format in playlist_formats:
+                    'url': manifest_url,
-                    format_index = format.get('index')
+                    'manifest_url': m3u8_url,
-                    for v in (m3u8_id, group_id, name):
+                    'language': media.get('LANGUAGE'),
-                        if v:
+                    'ext': ext,
-                            format_id.append(v)
+                    'protocol': entry_protocol,
-                    if format_index:
+                    'preference': preference,
-                        format_id.append(str(format_index))
+                    'quality': quality,
-                    f = {
+                    'vcodec': 'none' if media_type == 'AUDIO' else None,
-                        'format_id': '-'.join(format_id),
+                } for idx in _extract_m3u8_playlist_indices(manifest_url))
-                        'format_note': name,
-                        'format_index': format_index,
-                        'url': manifest_url,
-                        'manifest_url': m3u8_url,
-                        'language': media.get('LANGUAGE'),
-                        'ext': ext,
-                        'protocol': entry_protocol,
-                        'preference': preference,
-                        'quality': quality,
-                    }
-                    if media_type == 'AUDIO':
-                        f['vcodec'] = 'none'
-                    formats.append(f)
 
         def build_stream_name():
             # Despite specification does not mention NAME attribute for
@@ -2178,25 +2125,17 @@ class InfoExtractor(object):
                     or last_stream_inf.get('BANDWIDTH'), scale=1000)
                 manifest_url = format_url(line.strip())
 
-                playlist_formats = _extract_m3u8_playlist_formats(manifest_url, video_id=video_id,
+                for idx in _extract_m3u8_playlist_indices(manifest_url):
-                                                                  fatal=fatal, data=data, headers=headers)
+                    format_id = [m3u8_id, None, idx]
-
-                for frmt in playlist_formats:
-                    format_id = []
-                    if m3u8_id:
-                        format_id.append(m3u8_id)
-                    format_index = frmt.get('index')
-                    stream_name = build_stream_name()
                     # Bandwidth of live streams may differ over time thus making
                     # format_id unpredictable. So it's better to keep provided
                     # format_id intact.
                     if not live:
-                        format_id.append(stream_name if stream_name else '%d' % (tbr if tbr else len(formats)))
+                        stream_name = build_stream_name()
-                    if format_index:
+                        format_id[1] = stream_name if stream_name else '%d' % (tbr if tbr else len(formats))
-                        format_id.append(str(format_index))
                     f = {
-                        'format_id': '-'.join(format_id),
+                        'format_id': '-'.join(map(str, filter(None, format_id))),
-                        'format_index': format_index,
+                        'format_index': idx,
                         'url': manifest_url,
                         'manifest_url': m3u8_url,
                         'tbr': tbr,
@@ -3505,16 +3444,8 @@ class InfoExtractor(object):
         return ret
 
     @classmethod
-    def _merge_subtitles(cls, *dicts, **kwargs):
+    def _merge_subtitles(cls, *dicts, target=None):
         """ Merge subtitle dictionaries, language by language. """
-
-        target = (lambda target=None: target)(**kwargs)
-        # The above lambda extracts the keyword argument 'target' from kwargs
-        # while ensuring there are no stray ones. When Python 2 support
-        # is dropped, remove it and change the function signature to:
-        #
-        #     def _merge_subtitles(cls, *dicts, target=None):
-
         if target is None:
             target = {}
         for d in dicts:
@@ -3567,6 +3498,10 @@ class InfoExtractor(object):
             else 'public' if all_known
             else None)
 
+    def _configuration_arg(self, key):
+        return traverse_obj(
+            self._downloader.params, ('extractor_args', self.ie_key().lower(), key))
+
 
 class SearchInfoExtractor(InfoExtractor):
     """

yt_dlp/extractor/extractors.py

@@ -398,7 +398,11 @@ from .facebook import (
     FacebookIE,
     FacebookPluginsVideoIE,
 )
-from .fancode import FancodeVodIE
+from .fancode import (
+    FancodeVodIE,
+    FancodeLiveIE
+)
+
 from .faz import FazIE
 from .fc2 import (
     FC2IE,
@@ -455,7 +459,11 @@ from .frontendmasters import (
     FrontendMastersCourseIE
 )
 from .fujitv import FujiTVFODPlus7IE
-from .funimation import FunimationIE
+from .funimation import (
+    FunimationIE,
+    FunimationPageIE,
+    FunimationShowIE,
+)
 from .funk import FunkIE
 from .fusion import FusionIE
 from .gaia import GaiaIE
@@ -1060,6 +1068,10 @@ from .rcs import (
     RCSEmbedsIE,
     RCSVariousIE,
 )
+from .rcti import (
+    RCTIPlusIE,
+    RCTIPlusSeriesIE,
+)
 from .rds import RDSIE
 from .redbulltv import (
     RedBullTVIE,

yt_dlp/extractor/facebook.py

@@ -629,16 +629,11 @@ class FacebookIE(InfoExtractor):
 
         process_formats(formats)
 
+        description = self._html_search_meta('description', webpage, default=None)
         video_title = self._html_search_regex(
-            r'<h2\s+[^>]*class="uiHeaderTitle"[^>]*>([^<]*)</h2>', webpage,
+            (r'<h2\s+[^>]*class="uiHeaderTitle"[^>]*>([^<]*)</h2>',
-            'title', default=None)
+             r'(?s)<span class="fbPhotosPhotoCaption".*?id="fbPhotoPageCaption"><span class="hasCaption">(.*?)</span>'),
-        if not video_title:
+            webpage, 'title', default=None) or self._og_search_title(webpage, default=None) or description
-            video_title = self._html_search_regex(
-                r'(?s)<span class="fbPhotosPhotoCaption".*?id="fbPhotoPageCaption"><span class="hasCaption">(.*?)</span>',
-                webpage, 'alternative title', default=None)
-        if not video_title:
-            video_title = self._html_search_meta(
-                'description', webpage, 'title', default=None)
         if video_title:
             video_title = limit_length(video_title, 80)
         else:
@@ -662,6 +657,7 @@ class FacebookIE(InfoExtractor):
             'formats': formats,
             'uploader': uploader,
             'timestamp': timestamp,
+            'description': description,
             'thumbnail': thumbnail,
             'view_count': view_count,
             'subtitles': subtitles,

yt_dlp/extractor/fancode.py

@@ -7,7 +7,8 @@ from ..compat import compat_str
 from ..utils import (
     parse_iso8601,
     ExtractorError,
-    try_get
+    try_get,
+    mimetype2ext
 )
 
 
@@ -38,16 +39,63 @@ class FancodeVodIE(InfoExtractor):
         'only_matching': True,
     }]
 
+    _ACCESS_TOKEN = None
+    _NETRC_MACHINE = 'fancode'
+
+    _LOGIN_HINT = 'Use "--user refresh --password <refresh_token>" to login using a refresh token'
+
+    headers = {
+        'content-type': 'application/json',
+        'origin': 'https://fancode.com',
+        'referer': 'https://fancode.com',
+    }
+
+    def _login(self):
+        # Access tokens are shortlived, so get them using the refresh token.
+        username, password = self._get_login_info()
+        if username == 'refresh' and password is not None:
+            self.report_login()
+            data = '''{
+                "query":"mutation RefreshToken($refreshToken: String\\u0021) { refreshToken(refreshToken: $refreshToken) { accessToken }}",
+                "variables":{
+                    "refreshToken":"%s"
+                },
+                "operationName":"RefreshToken"
+            }''' % password
+
+            token_json = self.download_gql('refresh token', data, "Getting the Access token")
+            self._ACCESS_TOKEN = try_get(token_json, lambda x: x['data']['refreshToken']['accessToken'])
+            if self._ACCESS_TOKEN is None:
+                self.report_warning('Failed to get Access token')
+            else:
+                self.headers.update({'Authorization': 'Bearer %s' % self._ACCESS_TOKEN})
+        elif username is not None:
+            self.report_warning(f'Login using username and password is not currently supported. {self._LOGIN_HINT}')
+
+    def _real_initialize(self):
+        self._login()
+
+    def _check_login_required(self, is_available, is_premium):
+        msg = None
+        if is_premium and self._ACCESS_TOKEN is None:
+            msg = f'This video is only available for registered users. {self._LOGIN_HINT}'
+        elif not is_available and self._ACCESS_TOKEN is not None:
+            msg = 'This video isn\'t available to the current logged in account'
+        if msg:
+            self.raise_login_required(msg, metadata_available=True, method=None)
+
+    def download_gql(self, variable, data, note, fatal=False, headers=headers):
+        return self._download_json(
+            'https://www.fancode.com/graphql', variable,
+            data=data.encode(), note=note,
+            headers=headers, fatal=fatal)
+
     def _real_extract(self, url):
 
         BRIGHTCOVE_URL_TEMPLATE = 'https://players.brightcove.net/%s/default_default/index.html?videoId=%s'
-
         video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-        brightcove_user_id = self._html_search_regex(
-            r'(?:https?://)?players\.brightcove\.net/(\d+)/default_default/index(?:\.min)?\.js',
-            webpage, 'user id')
 
+        brightcove_user_id = '6008340455001'
         data = '''{
             "query":"query Video($id: Int\\u0021, $filter: SegmentFilter) { media(id: $id, filter: $filter) { id contentId title contentId publishedTime totalViews totalUpvotes provider thumbnail { src } mediaSource {brightcove } duration isPremium isUserEntitled tags duration }}",
             "variables":{
@@ -57,15 +105,9 @@ class FancodeVodIE(InfoExtractor):
                 }
             },
             "operationName":"Video"
-            }''' % video_id
+        }''' % video_id
 
-        metadata_json = self._download_json(
+        metadata_json = self.download_gql(video_id, data, note='Downloading metadata')
-            'https://www.fancode.com/graphql', video_id, data=data.encode(), note='Downloading metadata',
-            headers={
-                'content-type': 'application/json',
-                'origin': 'https://fancode.com',
-                'referer': url,
-            })
 
         media = try_get(metadata_json, lambda x: x['data']['media'], dict) or {}
         brightcove_video_id = try_get(media, lambda x: x['mediaSource']['brightcove'], compat_str)
@@ -74,8 +116,8 @@ class FancodeVodIE(InfoExtractor):
             raise ExtractorError('Unable to extract brightcove Video ID')
 
         is_premium = media.get('isPremium')
-        if is_premium:
+
-            self.report_warning('this video requires a premium account', video_id)
+        self._check_login_required(media.get('isUserEntitled'), is_premium)
 
         return {
             '_type': 'url_transparent',
@@ -89,3 +131,57 @@ class FancodeVodIE(InfoExtractor):
             'release_timestamp': parse_iso8601(media.get('publishedTime')),
             'availability': self._availability(needs_premium=is_premium),
         }
+
+
+class FancodeLiveIE(FancodeVodIE):
+    IE_NAME = 'fancode:live'
+
+    _VALID_URL = r'https?://(www\.)?fancode\.com/match/(?P<id>[0-9]+).+'
+
+    _TESTS = [{
+        'url': 'https://fancode.com/match/35328/cricket-fancode-ecs-hungary-2021-bub-vs-blb?slug=commentary',
+        'info_dict': {
+            'id': '35328',
+            'ext': 'mp4',
+            'title': 'BUB vs BLB',
+            "timestamp": 1624863600,
+            'is_live': True,
+            'upload_date': '20210628',
+        },
+        'skip': 'Ended'
+    }, {
+        'url': 'https://fancode.com/match/35328/',
+        'only_matching': True,
+    }, {
+        'url': 'https://fancode.com/match/35567?slug=scorecard',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+
+        id = self._match_id(url)
+        data = '''{
+            "query":"query MatchResponse($id: Int\\u0021, $isLoggedIn: Boolean\\u0021) { match: matchWithScores(id: $id) { id matchDesc mediaId videoStreamId videoStreamUrl { ...VideoSource } liveStreams { videoStreamId videoStreamUrl { ...VideoSource } contentId } name startTime streamingStatus isPremium isUserEntitled @include(if: $isLoggedIn) status metaTags bgImage { src } sport { name slug } tour { id name } squads { name shortName } liveStreams { contentId } mediaId }}fragment VideoSource on VideoSource { title description posterUrl url deliveryType playerType}",
+            "variables":{
+                "id":%s,
+                "isLoggedIn":true
+            },
+            "operationName":"MatchResponse"
+        }''' % id
+
+        info_json = self.download_gql(id, data, "Info json")
+
+        match_info = try_get(info_json, lambda x: x['data']['match'])
+
+        if match_info.get('status') != "LIVE":
+            raise ExtractorError('The stream can\'t be accessed', expected=True)
+        self._check_login_required(match_info.get('isUserEntitled'), True)  # all live streams are premium only
+
+        return {
+            'id': id,
+            'title': match_info.get('name'),
+            'formats': self._extract_akamai_formats(try_get(match_info, lambda x: x['videoStreamUrl']['url']), id),
+            'ext': mimetype2ext(try_get(match_info, lambda x: x['videoStreamUrl']['deliveryType'])),
+            'is_live': True,
+            'release_timestamp': parse_iso8601(match_info.get('startTime'))
+        }

yt_dlp/extractor/funimation.py

@@ -2,60 +2,124 @@
 from __future__ import unicode_literals
 
 import random
+import re
 import string
 
 from .common import InfoExtractor
 from ..compat import compat_HTTPError
 from ..utils import (
     determine_ext,
+    dict_get,
     int_or_none,
     js_to_json,
+    str_or_none,
+    try_get,
     urlencode_postdata,
-    urljoin,
     ExtractorError,
 )
 
 
-class FunimationIE(InfoExtractor):
+class FunimationPageIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?funimation(?:\.com|now\.uk)/(?:[^/]+/)?shows/[^/]+/(?P<id>[^/?#&]+)'
+    IE_NAME = 'funimation:page'
-
+    _VALID_URL = r'(?P<origin>https?://(?:www\.)?funimation(?:\.com|now\.uk))/(?P<lang>[^/]+/)?(?P<path>shows/(?P<id>[^/]+/[^/?#&]+).*$)'
-    _NETRC_MACHINE = 'funimation'
-    _TOKEN = None
 
     _TESTS = [{
-        'url': 'https://www.funimation.com/shows/hacksign/role-play/',
-        'info_dict': {
-            'id': '91144',
-            'display_id': 'role-play',
-            'ext': 'mp4',
-            'title': '.hack//SIGN - Role Play',
-            'description': 'md5:b602bdc15eef4c9bbb201bb6e6a4a2dd',
-            'thumbnail': r're:https?://.*\.jpg',
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
-    }, {
         'url': 'https://www.funimation.com/shows/attack-on-titan-junior-high/broadcast-dub-preview/',
         'info_dict': {
-            'id': '210051',
+            'id': '210050',
-            'display_id': 'broadcast-dub-preview',
             'ext': 'mp4',
-            'title': 'Attack on Titan: Junior High - Broadcast Dub Preview',
+            'title': 'Broadcast Dub Preview',
-            'thumbnail': r're:https?://.*\.(?:jpg|png)',
+            # Other metadata is tested in FunimationIE
         },
         'params': {
-            # m3u8 download
+            'skip_download': 'm3u8',
-            'skip_download': True,
         },
+        'add_ie': ['Funimation'],
     }, {
-        'url': 'https://www.funimationnow.uk/shows/puzzle-dragons-x/drop-impact/simulcast/',
+        # Not available in US
+        'url': 'https://www.funimation.com/shows/hacksign/role-play/',
         'only_matching': True,
     }, {
         # with lang code
         'url': 'https://www.funimation.com/en/shows/hacksign/role-play/',
         'only_matching': True,
+    }, {
+        'url': 'https://www.funimationnow.uk/shows/puzzle-dragons-x/drop-impact/simulcast/',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        mobj = re.match(self._VALID_URL, url)
+        display_id = mobj.group('id').replace('/', '_')
+        if not mobj.group('lang'):
+            url = '%s/en/%s' % (mobj.group('origin'), mobj.group('path'))
+
+        webpage = self._download_webpage(url, display_id)
+        title_data = self._parse_json(self._search_regex(
+            r'TITLE_DATA\s*=\s*({[^}]+})',
+            webpage, 'title data', default=''),
+            display_id, js_to_json, fatal=False) or {}
+
+        video_id = (
+            title_data.get('id')
+            or self._search_regex(
+                (r"KANE_customdimensions.videoID\s*=\s*'(\d+)';", r'<iframe[^>]+src="/player/(\d+)'),
+                webpage, 'video_id', default=None)
+            or self._search_regex(
+                r'/player/(\d+)',
+                self._html_search_meta(['al:web:url', 'og:video:url', 'og:video:secure_url'], webpage, fatal=True),
+                'video id'))
+        return self.url_result(f'https://www.funimation.com/player/{video_id}', FunimationIE.ie_key(), video_id)
+
+
+class FunimationIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?funimation\.com/player/(?P<id>\d+)'
+
+    _NETRC_MACHINE = 'funimation'
+    _TOKEN = None
+
+    _TESTS = [{
+        'url': 'https://www.funimation.com/player/210051',
+        'info_dict': {
+            'id': '210050',
+            'display_id': 'broadcast-dub-preview',
+            'ext': 'mp4',
+            'title': 'Broadcast Dub Preview',
+            'thumbnail': r're:https?://.*\.(?:jpg|png)',
+            'episode': 'Broadcast Dub Preview',
+            'episode_id': '210050',
+            'season': 'Extras',
+            'season_id': '166038',
+            'season_number': 99,
+            'series': 'Attack on Titan: Junior High',
+            'description': '',
+            'duration': 154,
+        },
+        'params': {
+            'skip_download': 'm3u8',
+        },
+    }, {
+        'note': 'player_id should be extracted with the relevent compat-opt',
+        'url': 'https://www.funimation.com/player/210051',
+        'info_dict': {
+            'id': '210051',
+            'display_id': 'broadcast-dub-preview',
+            'ext': 'mp4',
+            'title': 'Broadcast Dub Preview',
+            'thumbnail': r're:https?://.*\.(?:jpg|png)',
+            'episode': 'Broadcast Dub Preview',
+            'episode_id': '210050',
+            'season': 'Extras',
+            'season_id': '166038',
+            'season_number': 99,
+            'series': 'Attack on Titan: Junior High',
+            'description': '',
+            'duration': 154,
+        },
+        'params': {
+            'skip_download': 'm3u8',
+            'compat_opts': ['seperate-video-versions'],
+        },
     }]
 
     def _login(self):
@@ -79,100 +143,184 @@ class FunimationIE(InfoExtractor):
     def _real_initialize(self):
         self._login()
 
+    @staticmethod
+    def _get_experiences(episode):
+        for lang, lang_data in episode.get('languages', {}).items():
+            for video_data in lang_data.values():
+                for version, f in video_data.items():
+                    yield lang, version.title(), f
+
+    def _get_episode(self, webpage, experience_id=None, episode_id=None, fatal=True):
+        ''' Extract the episode, season and show objects given either episode/experience id '''
+        show = self._parse_json(
+            self._search_regex(
+                r'show\s*=\s*({.+?})\s*;', webpage, 'show data', fatal=fatal),
+            experience_id, transform_source=js_to_json, fatal=fatal) or []
+        for season in show.get('seasons', []):
+            for episode in season.get('episodes', []):
+                if episode_id is not None:
+                    if str(episode.get('episodePk')) == episode_id:
+                        return episode, season, show
+                    continue
+                for _, _, f in self._get_experiences(episode):
+                    if f.get('experienceId') == experience_id:
+                        return episode, season, show
+        if fatal:
+            raise ExtractorError('Unable to find episode information')
+        else:
+            self.report_warning('Unable to find episode information')
+        return {}, {}, {}
+
     def _real_extract(self, url):
-        display_id = self._match_id(url)
+        initial_experience_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
+        webpage = self._download_webpage(
+            url, initial_experience_id, note=f'Downloading player webpage for {initial_experience_id}')
+        episode, season, show = self._get_episode(webpage, experience_id=int(initial_experience_id))
+        episode_id = str(episode['episodePk'])
+        display_id = episode.get('slug') or episode_id
 
-        def _search_kane(name):
+        formats, subtitles, thumbnails, duration = [], {}, [], 0
-            return self._search_regex(
+        requested_languages, requested_versions = self._configuration_arg('language'), self._configuration_arg('version')
-                r"KANE_customdimensions\.%s\s*=\s*'([^']+)';" % name,
+        only_initial_experience = 'seperate-video-versions' in self.get_param('compat_opts', [])
-                webpage, name, default=None)
 
-        title_data = self._parse_json(self._search_regex(
+        for lang, version, fmt in self._get_experiences(episode):
-            r'TITLE_DATA\s*=\s*({[^}]+})',
+            experience_id = str(fmt['experienceId'])
-            webpage, 'title data', default=''),
+            if (only_initial_experience and experience_id != initial_experience_id
-            display_id, js_to_json, fatal=False) or {}
+                    or requested_languages and lang not in requested_languages
+                    or requested_versions and version not in requested_versions):
+                continue
+            thumbnails.append({'url': fmt.get('poster')})
+            duration = max(duration, fmt.get('duration', 0))
+            format_name = '%s %s (%s)' % (version, lang, experience_id)
+            self.extract_subtitles(
+                subtitles, experience_id, display_id=display_id, format_name=format_name,
+                episode=episode if experience_id == initial_experience_id else episode_id)
 
-        video_id = title_data.get('id') or self._search_regex([
-            r"KANE_customdimensions.videoID\s*=\s*'(\d+)';",
-            r'<iframe[^>]+src="/player/(\d+)',
-        ], webpage, 'video_id', default=None)
-        if not video_id:
-            player_url = self._html_search_meta([
-                'al:web:url',
-                'og:video:url',
-                'og:video:secure_url',
-            ], webpage, fatal=True)
-            video_id = self._search_regex(r'/player/(\d+)', player_url, 'video id')
-
-        title = episode = title_data.get('title') or _search_kane('videoTitle') or self._og_search_title(webpage)
-        series = _search_kane('showName')
-        if series:
-            title = '%s - %s' % (series, title)
-        description = self._html_search_meta(['description', 'og:description'], webpage, fatal=True)
-        subtitles = self.extract_subtitles(url, video_id, display_id)
-
-        try:
             headers = {}
             if self._TOKEN:
                 headers['Authorization'] = 'Token %s' % self._TOKEN
-            sources = self._download_json(
+            page = self._download_json(
-                'https://www.funimation.com/api/showexperience/%s/' % video_id,
+                'https://www.funimation.com/api/showexperience/%s/' % experience_id,
-                video_id, headers=headers, query={
+                display_id, headers=headers, expected_status=403, query={
                     'pinst_id': ''.join([random.choice(string.digits + string.ascii_letters) for _ in range(8)]),
-                })['items']
+                }, note=f'Downloading {format_name} JSON')
-        except ExtractorError as e:
+            sources = page.get('items') or []
-            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
+            if not sources:
-                error = self._parse_json(e.cause.read(), video_id)['errors'][0]
+                error = try_get(page, lambda x: x['errors'][0], dict)
-                raise ExtractorError('%s said: %s' % (
+                if error:
-                    self.IE_NAME, error.get('detail') or error.get('title')), expected=True)
+                    self.report_warning('%s said: Error %s - %s' % (
-            raise
+                        self.IE_NAME, error.get('code'), error.get('detail') or error.get('title')))
+                else:
+                    self.report_warning('No sources found for format')
 
-        formats = []
+            current_formats = []
-        for source in sources:
+            for source in sources:
-            source_url = source.get('src')
+                source_url = source.get('src')
-            if not source_url:
+                source_type = source.get('videoType') or determine_ext(source_url)
-                continue
+                if source_type == 'm3u8':
-            source_type = source.get('videoType') or determine_ext(source_url)
+                    current_formats.extend(self._extract_m3u8_formats(
-            if source_type == 'm3u8':
+                        source_url, display_id, 'mp4', m3u8_id='%s-%s' % (experience_id, 'hls'), fatal=False,
-                formats.extend(self._extract_m3u8_formats(
+                        note=f'Downloading {format_name} m3u8 information'))
-                    source_url, video_id, 'mp4',
+                else:
-                    m3u8_id='hls', fatal=False))
+                    current_formats.append({
-            else:
+                        'format_id': '%s-%s' % (experience_id, source_type),
-                formats.append({
+                        'url': source_url,
-                    'format_id': source_type,
+                    })
-                    'url': source_url,
+                for f in current_formats:
-                })
+                    # TODO: Convert language to code
+                    f.update({'language': lang, 'format_note': version})
+                formats.extend(current_formats)
+        self._remove_duplicate_formats(formats)
         self._sort_formats(formats)
 
         return {
-            'id': video_id,
+            'id': initial_experience_id if only_initial_experience else episode_id,
             'display_id': display_id,
-            'title': title,
+            'duration': duration,
-            'description': description,
+            'title': episode['episodeTitle'],
-            'thumbnail': self._og_search_thumbnail(webpage),
+            'description': episode.get('episodeSummary'),
-            'series': series,
+            'episode': episode.get('episodeTitle'),
-            'season_number': int_or_none(title_data.get('seasonNum') or _search_kane('season')),
+            'episode_number': int_or_none(episode.get('episodeId')),
-            'episode_number': int_or_none(title_data.get('episodeNum')),
+            'episode_id': episode_id,
-            'episode': episode,
+            'season': season.get('seasonTitle'),
-            'subtitles': subtitles,
+            'season_number': int_or_none(season.get('seasonId')),
-            'season_id': title_data.get('seriesId'),
+            'season_id': str_or_none(season.get('seasonPk')),
+            'series': show.get('showTitle'),
             'formats': formats,
+            'thumbnails': thumbnails,
+            'subtitles': subtitles,
         }
 
-    def _get_subtitles(self, url, video_id, display_id):
+    def _get_subtitles(self, subtitles, experience_id, episode, display_id, format_name):
-        player_url = urljoin(url, '/player/' + video_id)
+        if isinstance(episode, str):
-        player_page = self._download_webpage(player_url, display_id)
+            webpage = self._download_webpage(
-        text_tracks_json_string = self._search_regex(
+                f'https://www.funimation.com/player/{experience_id}', display_id,
-            r'"textTracks": (\[{.+?}\])',
+                fatal=False, note=f'Downloading player webpage for {format_name}')
-            player_page, 'subtitles data', default='')
+            episode, _, _ = self._get_episode(webpage, episode_id=episode, fatal=False)
-        text_tracks = self._parse_json(
+
-            text_tracks_json_string, display_id, js_to_json, fatal=False) or []
+        for _, version, f in self._get_experiences(episode):
-        subtitles = {}
+            for source in f.get('sources'):
-        for text_track in text_tracks:
+                for text_track in source.get('textTracks'):
-            url_element = {'url': text_track.get('src')}
+                    if not text_track.get('src'):
-            language = text_track.get('language')
+                        continue
-            if text_track.get('type') == 'CC':
+                    sub_type = text_track.get('type').upper()
-                language += '_CC'
+                    sub_type = sub_type if sub_type != 'FULL' else None
-            subtitles.setdefault(language, []).append(url_element)
+                    current_sub = {
+                        'url': text_track['src'],
+                        'name': ' '.join(filter(None, (version, text_track.get('label'), sub_type)))
+                    }
+                    lang = '_'.join(filter(None, (
+                        text_track.get('language', 'und'), version if version != 'Simulcast' else None, sub_type)))
+                    if current_sub not in subtitles.get(lang, []):
+                        subtitles.setdefault(lang, []).append(current_sub)
         return subtitles
+
+
+class FunimationShowIE(FunimationIE):
+    IE_NAME = 'funimation:show'
+    _VALID_URL = r'(?P<url>https?://(?:www\.)?funimation(?:\.com|now\.uk)/(?P<locale>[^/]+)?/?shows/(?P<id>[^/?#&]+))/?(?:[?#]|$)'
+
+    _TESTS = [{
+        'url': 'https://www.funimation.com/en/shows/sk8-the-infinity',
+        'info_dict': {
+            'id': 1315000,
+            'title': 'SK8 the Infinity'
+        },
+        'playlist_count': 13,
+        'params': {
+            'skip_download': True,
+        },
+    }, {
+        # without lang code
+        'url': 'https://www.funimation.com/shows/ouran-high-school-host-club/',
+        'info_dict': {
+            'id': 39643,
+            'title': 'Ouran High School Host Club'
+        },
+        'playlist_count': 26,
+        'params': {
+            'skip_download': True,
+        },
+    }]
+
+    def _real_extract(self, url):
+        base_url, locale, display_id = re.match(self._VALID_URL, url).groups()
+
+        show_info = self._download_json(
+            'https://title-api.prd.funimationsvc.com/v2/shows/%s?region=US&deviceType=web&locale=%s'
+            % (display_id, locale or 'en'), display_id)
+        items = self._download_json(
+            'https://prod-api-funimationnow.dadcdigital.com/api/funimation/episodes/?limit=99999&title_id=%s'
+            % show_info.get('id'), display_id).get('items')
+        vod_items = map(lambda k: dict_get(k, ('mostRecentSvod', 'mostRecentAvod')).get('item'), items)
+
+        return {
+            '_type': 'playlist',
+            'id': show_info['id'],
+            'title': show_info['name'],
+            'entries': [
+                self.url_result(
+                    '%s/%s' % (base_url, vod_item.get('episodeSlug')), FunimationPageIE.ie_key(),
+                    vod_item.get('episodeId'), vod_item.get('episodeName'))
+                for vod_item in sorted(vod_items, key=lambda x: x.get('episodeOrder'))],
+        }

yt_dlp/extractor/mtv.py

@@ -249,6 +249,7 @@ class MTVServicesInfoExtractor(InfoExtractor):
             if info:
                 entries.append(info)
 
+        # TODO: should be multi-video
         return self.playlist_result(
             entries, playlist_title=title, playlist_description=description)
 

yt_dlp/extractor/peertube.py

@@ -569,15 +569,15 @@ class PeerTubeIE(InfoExtractor):
             formats.append(f)
         self._sort_formats(formats)
 
-        full_description = self._call_api(
+        description = video.get('description')
-            host, video_id, 'description', note='Downloading description JSON',
+        if len(description) >= 250:
-            fatal=False)
+            # description is shortened
+            full_description = self._call_api(
+                host, video_id, 'description', note='Downloading description JSON',
+                fatal=False)
 
-        description = None
+            if isinstance(full_description, dict):
-        if isinstance(full_description, dict):
+                description = str_or_none(full_description.get('description')) or description
-            description = str_or_none(full_description.get('description'))
-        if not description:
-            description = video.get('description')
 
         subtitles = self.extract_subtitles(host, video_id)
 

yt_dlp/extractor/periscope.py

@@ -12,6 +12,10 @@ from ..utils import (
 
 
 class PeriscopeBaseIE(InfoExtractor):
+    _M3U8_HEADERS = {
+        'Referer': 'https://www.periscope.tv/'
+    }
+
     def _call_api(self, method, query, item_id):
         return self._download_json(
             'https://api.periscope.tv/api/v2/%s' % method,
@@ -54,9 +58,11 @@ class PeriscopeBaseIE(InfoExtractor):
             m3u8_url, video_id, 'mp4',
             entry_protocol='m3u8_native'
             if state in ('ended', 'timed_out') else 'm3u8',
-            m3u8_id=format_id, fatal=fatal)
+            m3u8_id=format_id, fatal=fatal, headers=self._M3U8_HEADERS)
         if len(m3u8_formats) == 1:
             self._add_width_and_height(m3u8_formats[0], width, height)
+        for f in m3u8_formats:
+            f.setdefault('http_headers', {}).update(self._M3U8_HEADERS)
         return m3u8_formats
 
 

yt_dlp/extractor/plutotv.py

@@ -19,7 +19,7 @@ from ..utils import (
 
 
 class PlutoTVIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?pluto\.tv/on-demand/(?P<video_type>movies|series)/(?P<slug>.*)/?$'
+    _VALID_URL = r'https?://(?:www\.)?pluto\.tv(?:/en)?/on-demand/(?P<video_type>movies|series)/(?P<slug>.*)/?$'
     _INFO_URL = 'https://service-vod.clusters.pluto.tv/v3/vod/slugs/'
     _INFO_QUERY_PARAMS = {
         'appName': 'web',
@@ -48,24 +48,21 @@ class PlutoTVIE(InfoExtractor):
                 'episode_number': 3,
                 'duration': 3600,
             }
-        },
+        }, {
-        {
             'url': 'https://pluto.tv/on-demand/series/i-love-money/season/1/',
             'playlist_count': 11,
             'info_dict': {
                 'id': '5de6c582e9379ae4912dedbd',
                 'title': 'I Love Money - Season 1',
             }
-        },
+        }, {
-        {
             'url': 'https://pluto.tv/on-demand/series/i-love-money/',
             'playlist_count': 26,
             'info_dict': {
                 'id': '5de6c582e9379ae4912dedbd',
                 'title': 'I Love Money',
             }
-        },
+        }, {
-        {
             'url': 'https://pluto.tv/on-demand/movies/arrival-2015-1-1',
             'md5': '3cead001d317a018bf856a896dee1762',
             'info_dict': {
@@ -75,7 +72,10 @@ class PlutoTVIE(InfoExtractor):
                 'description': 'When mysterious spacecraft touch down across the globe, an elite team - led by expert translator Louise Banks (Academy Award® nominee Amy Adams) – races against time to decipher their intent.',
                 'duration': 9000,
             }
-        },
+        }, {
+            'url': 'https://pluto.tv/en/on-demand/series/manhunters-fugitive-task-force/seasons/1/episode/third-times-the-charm-1-1',
+            'only_matching': True,
+        }
     ]
 
     def _to_ad_free_formats(self, video_id, formats, subtitles):

yt_dlp/extractor/rcti.py

@@ -0,0 +1,242 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import itertools
+import re
+
+from .openload import PhantomJSwrapper
+
+from .common import InfoExtractor
+from ..utils import (
+    ExtractorError,
+    RegexNotFoundError,
+    strip_or_none,
+    try_get
+)
+
+
+class RCTIPlusBaseIE(InfoExtractor):
+    def _real_initialize(self):
+        self._AUTH_KEY = self._download_json(
+            'https://api.rctiplus.com/api/v1/visitor?platform=web',  # platform can be web, mweb, android, ios
+            None, 'Fetching authorization key')['data']['access_token']
+
+    def _call_api(self, url, video_id, note=None):
+        json = self._download_json(
+            url, video_id, note=note, headers={'Authorization': self._AUTH_KEY})
+        if json.get('status', {}).get('code', 0) != 0:
+            raise ExtractorError('%s said: %s' % (self.IE_NAME, json["status"]["message_client"]), cause=json)
+        return json.get('data'), json.get('meta')
+
+
+class RCTIPlusIE(RCTIPlusBaseIE):
+    _VALID_URL = r'https://www\.rctiplus\.com/programs/\d+?/.*?/(?P<type>episode|clip|extra)/(?P<id>\d+)/(?P<display_id>[^/?#&]+)'
+    _TESTS = [{
+        'url': 'https://www.rctiplus.com/programs/1259/kiko-untuk-lola/episode/22124/untuk-lola',
+        'md5': '56ed45affad45fa18d5592a1bc199997',
+        'info_dict': {
+            'id': 'v_e22124',
+            'title': 'Untuk Lola',
+            'display_id': 'untuk-lola',
+            'description': 'md5:2b809075c0b1e071e228ad6d13e41deb',
+            'ext': 'mp4',
+            'duration': 1400,
+            'timestamp': 1615978800,
+            'upload_date': '20210317',
+            'series': 'Kiko : Untuk Lola',
+            'season_number': 1,
+            'episode_number': 1,
+            'channel': 'RCTI',
+        },
+        'params': {
+            'fixup': 'never',
+        },
+    }, {  # Clip; Series title doesn't appear on metadata JSON
+        'url': 'https://www.rctiplus.com/programs/316/cahaya-terindah/clip/3921/make-a-wish',
+        'md5': 'd179b2ff356f0e91a53bcc6a4d8504f0',
+        'info_dict': {
+            'id': 'v_c3921',
+            'title': 'Make A Wish',
+            'display_id': 'make-a-wish',
+            'description': 'Make A Wish',
+            'ext': 'mp4',
+            'duration': 288,
+            'timestamp': 1571652600,
+            'upload_date': '20191021',
+            'series': 'Cahaya Terindah',
+            'channel': 'RCTI',
+        },
+        'params': {
+            'fixup': 'never',
+        },
+    }, {  # Extra
+        'url': 'https://www.rctiplus.com/programs/616/inews-malam/extra/9438/diungkapkan-melalui-surat-terbuka-ceo-ruangguru-belva-devara-mundur-dari-staf-khusus-presiden',
+        'md5': 'c48106afdbce609749f5e0c007d9278a',
+        'info_dict': {
+            'id': 'v_ex9438',
+            'title': 'md5:2ede828c0f8bde249e0912be150314ca',
+            'display_id': 'md5:62b8d4e9ff096db527a1ad797e8a9933',
+            'description': 'md5:2ede828c0f8bde249e0912be150314ca',
+            'ext': 'mp4',
+            'duration': 93,
+            'timestamp': 1587561540,
+            'upload_date': '20200422',
+            'series': 'iNews Malam',
+            'channel': 'INews',
+        },
+        'params': {
+            'format': 'bestvideo',
+        },
+    }]
+
+    def _search_auth_key(self, webpage):
+        try:
+            self._AUTH_KEY = self._search_regex(
+                r'\'Authorization\':"(?P<auth>[^"]+)"', webpage, 'auth-key')
+        except RegexNotFoundError:
+            pass
+
+    def _real_extract(self, url):
+        video_type, video_id, display_id = re.match(self._VALID_URL, url).groups()
+        webpage = self._download_webpage(url, display_id)
+        self._search_auth_key(webpage)
+
+        video_json = self._call_api(
+            'https://api.rctiplus.com/api/v1/%s/%s/url?appierid=.1' % (video_type, video_id), display_id, 'Downloading video URL JSON')[0]
+        video_url = video_json['url']
+        if 'akamaized' in video_url:
+            # Akamai's CDN requires a session to at least be made via Conviva's API
+            # TODO: Reverse-engineer Conviva's heartbeat code to avoid phantomJS
+            phantom = None
+            try:
+                phantom = PhantomJSwrapper(self)
+                phantom.get(url, webpage, display_id, note2='Initiating video session')
+            except ExtractorError:
+                self.report_warning('PhantomJS is highly recommended for this video, as it might load incredibly slowly otherwise.'
+                                    'You can also try opening the page in this device\'s browser first')
+
+        video_meta, meta_paths = self._call_api(
+            'https://api.rctiplus.com/api/v1/%s/%s' % (video_type, video_id), display_id, 'Downloading video metadata')
+
+        thumbnails, image_path = [], meta_paths.get('image_path', 'https://rstatic.akamaized.net/media/')
+        if video_meta.get('portrait_image'):
+            thumbnails.append({
+                'id': 'portrait_image',
+                'url': '%s%d%s' % (image_path, 2000, video_meta['portrait_image'])  # 2000px seems to be the highest resolution that can be given
+            })
+        if video_meta.get('landscape_image'):
+            thumbnails.append({
+                'id': 'landscape_image',
+                'url': '%s%d%s' % (image_path, 2000, video_meta['landscape_image'])
+            })
+
+        formats = self._extract_m3u8_formats(video_url, display_id, 'mp4', headers={'Referer': 'https://www.rctiplus.com/'})
+        for f in formats:
+            if 'akamaized' in f['url']:
+                f.setdefault('http_headers', {})['Referer'] = 'https://www.rctiplus.com/'  # Referer header is required for akamai CDNs
+
+        self._sort_formats(formats)
+
+        return {
+            'id': video_meta.get('product_id') or video_json.get('product_id'),
+            'title': video_meta.get('title') or video_json.get('content_name'),
+            'display_id': display_id,
+            'description': video_meta.get('summary'),
+            'timestamp': video_meta.get('release_date'),
+            'duration': video_meta.get('duration'),
+            'categories': [video_meta.get('genre')],
+            'average_rating': video_meta.get('star_rating'),
+            'series': video_meta.get('program_title') or video_json.get('program_title'),
+            'season_number': video_meta.get('season'),
+            'episode_number': video_meta.get('episode'),
+            'channel': video_json.get('tv_name'),
+            'channel_id': video_json.get('tv_id'),
+            'formats': formats,
+            'thumbnails': thumbnails
+        }
+
+
+class RCTIPlusSeriesIE(RCTIPlusBaseIE):
+    _VALID_URL = r'https://www\.rctiplus\.com/programs/(?P<id>\d+)/(?P<display_id>[^/?#&]+)(?:\W)*$'
+    _TESTS = [{
+        'url': 'https://www.rctiplus.com/programs/540/upin-ipin',
+        'playlist_mincount': 417,
+        'info_dict': {
+            'id': '540',
+            'title': 'Upin & Ipin',
+            'description': 'md5:22cc912381f389664416844e1ec4f86b',
+        },
+    }, {
+        'url': 'https://www.rctiplus.com/programs/540/upin-ipin/#',
+        'only_matching': True,
+    }]
+    _AGE_RATINGS = {  # Based off https://id.wikipedia.org/wiki/Sistem_rating_konten_televisi with additional ratings
+        'S-SU': 2,
+        'SU': 2,
+        'P': 2,
+        'A': 7,
+        'R': 13,
+        'R-R/1': 17,  # Labelled as 17+ despite being R
+        'D': 18,
+    }
+
+    def _entries(self, url, display_id=None, note='Downloading entries JSON', metadata={}):
+        total_pages = 0
+        try:
+            total_pages = self._call_api(
+                '%s&length=20&page=0' % url,
+                display_id, note)[1]['pagination']['total_page']
+        except ExtractorError as e:
+            if 'not found' in str(e):
+                return []
+            raise e
+        if total_pages <= 0:
+            return []
+
+        for page_num in range(1, total_pages + 1):
+            episode_list = self._call_api(
+                '%s&length=20&page=%s' % (url, page_num),
+                display_id, '%s page %s' % (note, page_num))[0] or []
+
+            for video_json in episode_list:
+                link = video_json['share_link']
+                url_res = self.url_result(link, 'RCTIPlus', video_json.get('product_id'), video_json.get('title'))
+                url_res.update(metadata)
+                yield url_res
+
+    def _real_extract(self, url):
+        series_id, display_id = re.match(self._VALID_URL, url).groups()
+
+        series_meta, meta_paths = self._call_api(
+            'https://api.rctiplus.com/api/v1/program/%s/detail' % series_id, display_id, 'Downloading series metadata')
+        metadata = {
+            'age_limit': try_get(series_meta, lambda x: self._AGE_RATINGS[x['age_restriction'][0]['code']])
+        }
+
+        cast = []
+        for star in series_meta.get('starring', []):
+            cast.append(strip_or_none(star.get('name')))
+        for star in series_meta.get('creator', []):
+            cast.append(strip_or_none(star.get('name')))
+        for star in series_meta.get('writer', []):
+            cast.append(strip_or_none(star.get('name')))
+        metadata['cast'] = cast
+
+        tags = []
+        for tag in series_meta.get('tag', []):
+            tags.append(strip_or_none(tag.get('name')))
+        metadata['tag'] = tags
+
+        entries = []
+        seasons_list = self._call_api(
+            'https://api.rctiplus.com/api/v1/program/%s/season' % series_id, display_id, 'Downloading seasons list JSON')[0]
+        for season in seasons_list:
+            entries.append(self._entries('https://api.rctiplus.com/api/v2/program/%s/episode?season=%s' % (series_id, season['season']),
+                                         display_id, 'Downloading season %s episode entries' % season['season'], metadata))
+
+        entries.append(self._entries('https://api.rctiplus.com/api/v2/program/%s/clip?content_id=0' % series_id,
+                                     display_id, 'Downloading clip entries', metadata))
+        entries.append(self._entries('https://api.rctiplus.com/api/v2/program/%s/extra?content_id=0' % series_id,
+                                     display_id, 'Downloading extra entries', metadata))
+
+        return self.playlist_result(itertools.chain(*entries), series_id, series_meta.get('title'), series_meta.get('summary'), **metadata)

yt_dlp/extractor/soundcloud.py

@@ -4,7 +4,7 @@ from __future__ import unicode_literals
 import itertools
 import re
 import json
-import random
+# import random
 
 from .common import (
     InfoExtractor,
@@ -164,23 +164,11 @@ class SoundcloudIE(InfoExtractor):
         },
         # downloadable song
         {
-            'url': 'https://soundcloud.com/oddsamples/bus-brakes',
+            'url': 'https://soundcloud.com/the80m/the-following',
-            'md5': '7624f2351f8a3b2e7cd51522496e7631',
+            'md5': '9ffcddb08c87d74fb5808a3c183a1d04',
             'info_dict': {
-                'id': '128590877',
+                'id': '343609555',
-                'ext': 'mp3',
+                'ext': 'wav',
-                'title': 'Bus Brakes',
-                'description': 'md5:0053ca6396e8d2fd7b7e1595ef12ab66',
-                'uploader': 'oddsamples',
-                'uploader_id': '73680509',
-                'timestamp': 1389232924,
-                'upload_date': '20140109',
-                'duration': 17.346,
-                'license': 'cc-by-sa',
-                'view_count': int,
-                'like_count': int,
-                'comment_count': int,
-                'repost_count': int,
             },
         },
         # private link, downloadable format
@@ -317,12 +305,13 @@ class SoundcloudIE(InfoExtractor):
                 raise
 
     def _real_initialize(self):
-        self._CLIENT_ID = self._downloader.cache.load('soundcloud', 'client_id') or "T5R4kgWS2PRf6lzLyIravUMnKlbIxQag"  # 'EXLwg5lHTO2dslU5EePe3xkw0m1h86Cd' # 'YUKXoArFcqrlQn9tfNHvvyfnDISj04zk'
+        self._CLIENT_ID = self._downloader.cache.load('soundcloud', 'client_id') or 'fXuVKzsVXlc6tzniWWS31etd7VHWFUuN'  # persistent `client_id`
         self._login()
 
-    _USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
+    _USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36'
     _API_AUTH_QUERY_TEMPLATE = '?client_id=%s'
     _API_AUTH_URL_PW = 'https://api-auth.soundcloud.com/web-auth/sign-in/password%s'
+    _API_VERIFY_AUTH_TOKEN = 'https://api-auth.soundcloud.com/connect/session%s'
     _access_token = None
     _HEADERS = {}
     _NETRC_MACHINE = 'soundcloud'
@@ -332,6 +321,23 @@ class SoundcloudIE(InfoExtractor):
         if username is None:
             return
 
+        if username == 'oauth' and password is not None:
+            self._access_token = password
+            query = self._API_AUTH_QUERY_TEMPLATE % self._CLIENT_ID
+            payload = {'session': {'access_token': self._access_token}}
+            token_verification = sanitized_Request(self._API_VERIFY_AUTH_TOKEN % query, json.dumps(payload).encode('utf-8'))
+            response = self._download_json(token_verification, None, note='Verifying login token...', fatal=False)
+            if response is not False:
+                self._HEADERS = {'Authorization': 'OAuth ' + self._access_token}
+                self.report_login()
+            else:
+                self.report_warning('Provided authorization token seems to be invalid. Continue as guest')
+        elif username is not None:
+            self.report_warning(
+                'Login using username and password is not currently supported. '
+                'Use "--user oauth --password <oauth_token>" to login using an oauth token')
+
+        r'''
         def genDevId():
             def genNumBlock():
                 return ''.join([str(random.randrange(10)) for i in range(6)])
@@ -358,6 +364,7 @@ class SoundcloudIE(InfoExtractor):
             self.report_warning('Unable to get access token, login may has failed')
         else:
             self._HEADERS = {'Authorization': 'OAuth ' + self._access_token}
+        '''
 
     # signature generation
     def sign(self, user, pw, clid):
@@ -370,9 +377,9 @@ class SoundcloudIE(InfoExtractor):
         b = 37
         k = 37
         c = 5
-        n = "0763ed7314c69015fd4a0dc16bbf4b90"  # _KEY
+        n = '0763ed7314c69015fd4a0dc16bbf4b90'  # _KEY
-        y = "8"  # _REV
+        y = '8'  # _REV
-        r = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"  # _USER_AGENT
+        r = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36'  # _USER_AGENT
         e = user  # _USERNAME
         t = clid  # _CLIENT_ID
 

yt_dlp/extractor/tbs.py

@@ -16,7 +16,7 @@ from ..utils import (
 
 
 class TBSIE(TurnerBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?(?P<site>tbs|tntdrama)\.com(?P<path>/(?:movies|shows/[^/]+/(?:clips|season-\d+/episode-\d+))/(?P<id>[^/?#]+))'
+    _VALID_URL = r'https?://(?:www\.)?(?P<site>tbs|tntdrama)\.com(?P<path>/(?:movies|watchtnt|shows/[^/]+/(?:clips|season-\d+/episode-\d+))/(?P<id>[^/?#]+))'
     _TESTS = [{
         'url': 'http://www.tntdrama.com/shows/the-alienist/clips/monster',
         'info_dict': {
@@ -45,7 +45,8 @@ class TBSIE(TurnerBaseIE):
         drupal_settings = self._parse_json(self._search_regex(
             r'<script[^>]+?data-drupal-selector="drupal-settings-json"[^>]*?>({.+?})</script>',
             webpage, 'drupal setting'), display_id)
-        video_data = next(v for v in drupal_settings['turner_playlist'] if v.get('url') == path)
+        isLive = 'watchtnt' in path
+        video_data = next(v for v in drupal_settings['turner_playlist'] if isLive or v.get('url') == path)
 
         media_id = video_data['mediaID']
         title = video_data['title']
@@ -56,7 +57,8 @@ class TBSIE(TurnerBaseIE):
             media_id, tokenizer_query, {
                 'url': url,
                 'site_name': site[:3].upper(),
-                'auth_required': video_data.get('authRequired') == '1',
+                'auth_required': video_data.get('authRequired') == '1' or isLive,
+                'is_live': isLive
             })
 
         thumbnails = []
@@ -85,5 +87,6 @@ class TBSIE(TurnerBaseIE):
             'season_number': int_or_none(video_data.get('season')),
             'episode_number': int_or_none(video_data.get('episode')),
             'thumbnails': thumbnails,
+            'is_live': isLive
         })
         return info

yt_dlp/extractor/turner.py

@@ -221,6 +221,7 @@ class TurnerBaseIE(AdobePassIE):
         }
 
     def _extract_ngtv_info(self, media_id, tokenizer_query, ap_data=None):
+        is_live = ap_data.get('is_live')
         streams_data = self._download_json(
             'http://medium.ngtv.io/media/%s/tv' % media_id,
             media_id)['media']['tv']
@@ -237,11 +238,11 @@ class TurnerBaseIE(AdobePassIE):
                     'http://token.ngtv.io/token/token_spe',
                     m3u8_url, media_id, ap_data or {}, tokenizer_query)
             formats.extend(self._extract_m3u8_formats(
-                m3u8_url, media_id, 'mp4', m3u8_id='hls', fatal=False))
+                m3u8_url, media_id, 'mp4', m3u8_id='hls', live=is_live, fatal=False))
 
             duration = float_or_none(stream_data.get('totalRuntime'))
 
-            if not chapters:
+            if not chapters and not is_live:
                 for chapter in stream_data.get('contentSegments', []):
                     start_time = float_or_none(chapter.get('start'))
                     chapter_duration = float_or_none(chapter.get('duration'))

yt_dlp/extractor/videa.py

@@ -12,6 +12,7 @@ from ..utils import (
     mimetype2ext,
     parse_codecs,
     update_url_query,
+    urljoin,
     xpath_element,
     xpath_text,
 )
@@ -19,6 +20,7 @@ from ..compat import (
     compat_b64decode,
     compat_ord,
     compat_struct_pack,
+    compat_urlparse,
 )
 
 
@@ -95,9 +97,13 @@ class VideaIE(InfoExtractor):
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        query = {'v': video_id}
+
-        player_page = self._download_webpage(
+        video_page = self._download_webpage(url, video_id)
-            'https://videa.hu/player', video_id, query=query)
+
+        player_url = self._search_regex(
+            r'<iframe.*?src="(/player\?[^"]+)"', video_page, 'player url')
+        player_url = urljoin(url, player_url)
+        player_page = self._download_webpage(player_url, video_id)
 
         nonce = self._search_regex(
             r'_xt\s*=\s*"([^"]+)"', player_page, 'nonce')
@@ -107,6 +113,7 @@ class VideaIE(InfoExtractor):
         for i in range(0, 32):
             result += s[i - (self._STATIC_SECRET.index(l[i]) - 31)]
 
+        query = compat_urlparse.parse_qs(compat_urlparse.urlparse(player_url).query)
         random_seed = ''.join(random.choice(string.ascii_letters + string.digits) for _ in range(8))
         query['_s'] = random_seed
         query['_t'] = result[:16]

yt_dlp/extractor/yahoo.py

@@ -22,6 +22,7 @@ from ..utils import (
 )
 
 from .brightcove import BrightcoveNewIE
+from .youtube import YoutubeIE
 
 
 class YahooIE(InfoExtractor):
@@ -38,6 +39,7 @@ class YahooIE(InfoExtractor):
             'timestamp': 1369812016,
             'upload_date': '20130529',
         },
+        'skip': 'No longer exists',
     }, {
         'url': 'https://screen.yahoo.com/community/community-sizzle-reel-203225340.html?format=embed',
         'md5': '7993e572fac98e044588d0b5260f4352',
@@ -50,6 +52,7 @@ class YahooIE(InfoExtractor):
             'timestamp': 1406838636,
             'upload_date': '20140731',
         },
+        'skip': 'Unfortunately, this video is not available in your region',
     }, {
         'url': 'https://uk.screen.yahoo.com/editor-picks/cute-raccoon-freed-drain-using-091756545.html',
         'md5': '71298482f7c64cbb7fa064e4553ff1c1',
@@ -61,7 +64,8 @@ class YahooIE(InfoExtractor):
             'duration': 97,
             'timestamp': 1414489862,
             'upload_date': '20141028',
-        }
+        },
+        'skip': 'No longer exists',
     }, {
         'url': 'http://news.yahoo.com/video/china-moses-crazy-blues-104538833.html',
         'md5': '88e209b417f173d86186bef6e4d1f160',
@@ -120,6 +124,7 @@ class YahooIE(InfoExtractor):
             'season_number': 6,
             'episode_number': 1,
         },
+        'skip': 'No longer exists',
     }, {
         # ytwnews://cavideo/
         'url': 'https://tw.video.yahoo.com/movie-tw/單車天使-中文版預-092316541.html',
@@ -156,7 +161,7 @@ class YahooIE(InfoExtractor):
                 'id': '352CFDOQrKg',
                 'ext': 'mp4',
                 'title': 'Kyndal Inskeep "Performs the Hell Out of" Sia\'s "Elastic Heart" - The Voice Knockouts 2019',
-                'description': 'md5:35b61e94c2ae214bc965ff4245f80d11',
+                'description': 'md5:7fe8e3d5806f96002e55f190d1d94479',
                 'uploader': 'The Voice',
                 'uploader_id': 'NBCTheVoice',
                 'upload_date': '20191029',
@@ -165,7 +170,7 @@ class YahooIE(InfoExtractor):
         'params': {
             'playlistend': 2,
         },
-        'expected_warnings': ['HTTP Error 404'],
+        'expected_warnings': ['HTTP Error 404', 'Ignoring subtitle tracks'],
     }, {
         'url': 'https://malaysia.news.yahoo.com/video/bystanders-help-ontario-policeman-bust-190932818.html',
         'only_matching': True,
@@ -280,12 +285,13 @@ class YahooIE(InfoExtractor):
         else:
             country = country.split('-')[0]
 
-        item = self._download_json(
+        items = self._download_json(
             'https://%s.yahoo.com/caas/content/article' % country, display_id,
             'Downloading content JSON metadata', query={
                 'url': url
-            })['items'][0]['data']['partnerData']
+            })['items'][0]
 
+        item = items['data']['partnerData']
         if item.get('type') != 'video':
             entries = []
 
@@ -299,9 +305,19 @@ class YahooIE(InfoExtractor):
             for e in (item.get('body') or []):
                 if e.get('type') == 'videoIframe':
                     iframe_url = e.get('url')
-                    if not iframe_url:
+                    if iframe_url:
-                        continue
+                        entries.append(self.url_result(iframe_url))
+
+            if item.get('type') == 'storywithleadvideo':
+                iframe_url = try_get(item, lambda x: x['meta']['player']['url'])
+                if iframe_url:
                     entries.append(self.url_result(iframe_url))
+                else:
+                    self.report_warning("Yahoo didn't provide an iframe url for this storywithleadvideo")
+
+            if items.get('markup'):
+                entries.extend(
+                    self.url_result(yt_url) for yt_url in YoutubeIE._extract_urls(items['markup']))
 
             return self.playlist_result(
                 entries, item.get('uuid'),

yt_dlp/extractor/youtube.py

@@ -3,6 +3,7 @@
 from __future__ import unicode_literals
 
 import calendar
+import copy
 import hashlib
 import itertools
 import json
@@ -294,13 +295,148 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
         if not self._login():
             return
 
-    _YT_WEB_CLIENT_VERSION = '2.20210407.08.00'
-    _YT_INNERTUBE_API_KEY = 'AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8'
     _YT_INITIAL_DATA_RE = r'(?:window\s*\[\s*["\']ytInitialData["\']\s*\]|ytInitialData)\s*=\s*({.+?})\s*;'
     _YT_INITIAL_PLAYER_RESPONSE_RE = r'ytInitialPlayerResponse\s*=\s*({.+?})\s*;'
     _YT_INITIAL_BOUNDARY_RE = r'(?:var\s+meta|</script|\n)'
 
-    def _generate_sapisidhash_header(self):
+    _YT_DEFAULT_YTCFGS = {
+        'WEB': {
+            'INNERTUBE_API_VERSION': 'v1',
+            'INNERTUBE_CLIENT_NAME': 'WEB',
+            'INNERTUBE_CLIENT_VERSION': '2.20210622.10.00',
+            'INNERTUBE_API_KEY': 'AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8',
+            'INNERTUBE_CONTEXT': {
+                'client': {
+                    'clientName': 'WEB',
+                    'clientVersion': '2.20210622.10.00',
+                    'hl': 'en',
+                }
+            },
+            'INNERTUBE_CONTEXT_CLIENT_NAME': 1
+        },
+        'WEB_REMIX': {
+            'INNERTUBE_API_VERSION': 'v1',
+            'INNERTUBE_CLIENT_NAME': 'WEB_REMIX',
+            'INNERTUBE_CLIENT_VERSION': '1.20210621.00.00',
+            'INNERTUBE_API_KEY': 'AIzaSyC9XL3ZjWddXya6X74dJoCTL-WEYFDNX30',
+            'INNERTUBE_CONTEXT': {
+                'client': {
+                    'clientName': 'WEB_REMIX',
+                    'clientVersion': '1.20210621.00.00',
+                    'hl': 'en',
+                }
+            },
+            'INNERTUBE_CONTEXT_CLIENT_NAME': 67
+        },
+        'WEB_EMBEDDED_PLAYER': {
+            'INNERTUBE_API_VERSION': 'v1',
+            'INNERTUBE_CLIENT_NAME': 'WEB_EMBEDDED_PLAYER',
+            'INNERTUBE_CLIENT_VERSION': '1.20210620.0.1',
+            'INNERTUBE_API_KEY': 'AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8',
+            'INNERTUBE_CONTEXT': {
+                'client': {
+                    'clientName': 'WEB_EMBEDDED_PLAYER',
+                    'clientVersion': '1.20210620.0.1',
+                    'hl': 'en',
+                }
+            },
+            'INNERTUBE_CONTEXT_CLIENT_NAME': 56
+        },
+        'ANDROID': {
+            'INNERTUBE_API_VERSION': 'v1',
+            'INNERTUBE_CLIENT_NAME': 'ANDROID',
+            'INNERTUBE_CLIENT_VERSION': '16.20',
+            'INNERTUBE_API_KEY': 'AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8',
+            'INNERTUBE_CONTEXT': {
+                'client': {
+                    'clientName': 'ANDROID',
+                    'clientVersion': '16.20',
+                    'hl': 'en',
+                }
+            },
+            'INNERTUBE_CONTEXT_CLIENT_NAME': 'ANDROID'
+        },
+        'ANDROID_EMBEDDED_PLAYER': {
+            'INNERTUBE_API_VERSION': 'v1',
+            'INNERTUBE_CLIENT_NAME': 'ANDROID_EMBEDDED_PLAYER',
+            'INNERTUBE_CLIENT_VERSION': '16.20',
+            'INNERTUBE_API_KEY': 'AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8',
+            'INNERTUBE_CONTEXT': {
+                'client': {
+                    'clientName': 'ANDROID_EMBEDDED_PLAYER',
+                    'clientVersion': '16.20',
+                    'hl': 'en',
+                }
+            },
+            'INNERTUBE_CONTEXT_CLIENT_NAME': 'ANDROID_EMBEDDED_PLAYER'
+        },
+        'ANDROID_MUSIC': {
+            'INNERTUBE_API_VERSION': 'v1',
+            'INNERTUBE_CLIENT_NAME': 'ANDROID_MUSIC',
+            'INNERTUBE_CLIENT_VERSION': '4.32',
+            'INNERTUBE_API_KEY': 'AIzaSyC9XL3ZjWddXya6X74dJoCTL-WEYFDNX30',
+            'INNERTUBE_CONTEXT': {
+                'client': {
+                    'clientName': 'ANDROID_MUSIC',
+                    'clientVersion': '4.32',
+                    'hl': 'en',
+                }
+            },
+            'INNERTUBE_CONTEXT_CLIENT_NAME': 'ANDROID_MUSIC'
+        }
+    }
+
+    _YT_DEFAULT_INNERTUBE_HOSTS = {
+        'DIRECT': 'youtubei.googleapis.com',
+        'WEB': 'www.youtube.com',
+        'WEB_REMIX': 'music.youtube.com',
+        'ANDROID_MUSIC': 'music.youtube.com'
+    }
+
+    def _get_default_ytcfg(self, client='WEB'):
+        if client in self._YT_DEFAULT_YTCFGS:
+            return copy.deepcopy(self._YT_DEFAULT_YTCFGS[client])
+        self.write_debug(f'INNERTUBE default client {client} does not exist - falling back to WEB client.')
+        return copy.deepcopy(self._YT_DEFAULT_YTCFGS['WEB'])
+
+    def _get_innertube_host(self, client='WEB'):
+        return dict_get(self._YT_DEFAULT_INNERTUBE_HOSTS, (client, 'WEB'))
+
+    def _ytcfg_get_safe(self, ytcfg, getter, expected_type=None, default_client='WEB'):
+        # try_get but with fallback to default ytcfg client values when present
+        _func = lambda y: try_get(y, getter, expected_type)
+        return _func(ytcfg) or _func(self._get_default_ytcfg(default_client))
+
+    def _extract_client_name(self, ytcfg, default_client='WEB'):
+        return self._ytcfg_get_safe(ytcfg, lambda x: x['INNERTUBE_CLIENT_NAME'], compat_str, default_client)
+
+    def _extract_client_version(self, ytcfg, default_client='WEB'):
+        return self._ytcfg_get_safe(ytcfg, lambda x: x['INNERTUBE_CLIENT_VERSION'], compat_str, default_client)
+
+    def _extract_api_key(self, ytcfg=None, default_client='WEB'):
+        return self._ytcfg_get_safe(ytcfg, lambda x: x['INNERTUBE_API_KEY'], compat_str, default_client)
+
+    def _extract_context(self, ytcfg=None, default_client='WEB'):
+        _get_context = lambda y: try_get(y, lambda x: x['INNERTUBE_CONTEXT'], dict)
+        context = _get_context(ytcfg)
+        if context:
+            return context
+
+        context = _get_context(self._get_default_ytcfg(default_client))
+        if not ytcfg:
+            return context
+
+        # Recreate the client context (required)
+        context['client'].update({
+            'clientVersion': self._extract_client_version(ytcfg, default_client),
+            'clientName': self._extract_client_name(ytcfg, default_client),
+        })
+        visitor_data = try_get(ytcfg, lambda x: x['VISITOR_DATA'], compat_str)
+        if visitor_data:
+            context['client']['visitorData'] = visitor_data
+        return context
+
+    def _generate_sapisidhash_header(self, origin='https://www.youtube.com'):
         # Sometimes SAPISID cookie isn't present but __Secure-3PAPISID is.
         # See: https://github.com/yt-dlp/yt-dlp/issues/393
         yt_cookies = self._get_cookies('https://www.youtube.com')
@@ -315,28 +451,25 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
                 '.youtube.com', 'SAPISID', sapisid_cookie.value, secure=True, expire_time=time_now + 3600)
         # SAPISIDHASH algorithm from https://stackoverflow.com/a/32065323
         sapisidhash = hashlib.sha1(
-            f'{time_now} {sapisid_cookie.value} https://www.youtube.com'.encode('utf-8')).hexdigest()
+            f'{time_now} {sapisid_cookie.value} {origin}'.encode('utf-8')).hexdigest()
         return f'SAPISIDHASH {time_now}_{sapisidhash}'
 
     def _call_api(self, ep, query, video_id, fatal=True, headers=None,
                   note='Downloading API JSON', errnote='Unable to download API page',
-                  context=None, api_key=None):
+                  context=None, api_key=None, api_hostname=None, default_client='WEB'):
 
-        data = {'context': context} if context else {'context': self._extract_context()}
+        data = {'context': context} if context else {'context': self._extract_context(default_client=default_client)}
         data.update(query)
-        real_headers = self._generate_api_headers()
+        real_headers = self._generate_api_headers(client=default_client)
         real_headers.update({'content-type': 'application/json'})
         if headers:
             real_headers.update(headers)
         return self._download_json(
-            'https://www.youtube.com/youtubei/v1/%s' % ep,
+            'https://%s/youtubei/v1/%s' % (api_hostname or self._get_innertube_host(default_client), ep),
             video_id=video_id, fatal=fatal, note=note, errnote=errnote,
             data=json.dumps(data).encode('utf8'), headers=real_headers,
             query={'key': api_key or self._extract_api_key()})
 
-    def _extract_api_key(self, ytcfg=None):
-        return try_get(ytcfg, lambda x: x['INNERTUBE_API_KEY'], compat_str) or self._YT_INNERTUBE_API_KEY
-
     def _extract_yt_initial_data(self, video_id, webpage):
         return self._parse_json(
             self._search_regex(
@@ -378,46 +511,118 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
                 r'ytcfg\.set\s*\(\s*({.+?})\s*\)\s*;', webpage, 'ytcfg',
                 default='{}'), video_id, fatal=False) or {}
 
-    def __extract_client_version(self, ytcfg):
+    def _generate_api_headers(self, ytcfg=None, identity_token=None, account_syncid=None,
-        return try_get(ytcfg, lambda x: x['INNERTUBE_CLIENT_VERSION'], compat_str) or self._YT_WEB_CLIENT_VERSION
+                              visitor_data=None, api_hostname=None, client='WEB'):
-
+        origin = 'https://' + (api_hostname if api_hostname else self._get_innertube_host(client))
-    def _extract_context(self, ytcfg=None):
-        context = try_get(ytcfg, lambda x: x['INNERTUBE_CONTEXT'], dict)
-        if context:
-            return context
-
-        # Recreate the client context (required)
-        client_version = self.__extract_client_version(ytcfg)
-        client_name = try_get(ytcfg, lambda x: x['INNERTUBE_CLIENT_NAME'], compat_str) or 'WEB'
-        context = {
-            'client': {
-                'clientName': client_name,
-                'clientVersion': client_version,
-            }
-        }
-        visitor_data = try_get(ytcfg, lambda x: x['VISITOR_DATA'], compat_str)
-        if visitor_data:
-            context['client']['visitorData'] = visitor_data
-        return context
-
-    def _generate_api_headers(self, ytcfg=None, identity_token=None, account_syncid=None, visitor_data=None):
         headers = {
-            'X-YouTube-Client-Name': '1',
+            'X-YouTube-Client-Name': compat_str(
-            'X-YouTube-Client-Version': self.__extract_client_version(ytcfg),
+                self._ytcfg_get_safe(ytcfg, lambda x: x['INNERTUBE_CONTEXT_CLIENT_NAME'], default_client=client)),
+            'X-YouTube-Client-Version': self._extract_client_version(ytcfg, client),
+            'Origin': origin
         }
         if identity_token:
-            headers['x-youtube-identity-token'] = identity_token
+            headers['X-Youtube-Identity-Token'] = identity_token
         if account_syncid:
             headers['X-Goog-PageId'] = account_syncid
             headers['X-Goog-AuthUser'] = 0
         if visitor_data:
-            headers['x-goog-visitor-id'] = visitor_data
+            headers['X-Goog-Visitor-Id'] = visitor_data
-        auth = self._generate_sapisidhash_header()
+        auth = self._generate_sapisidhash_header(origin)
         if auth is not None:
             headers['Authorization'] = auth
-            headers['X-Origin'] = 'https://www.youtube.com'
+            headers['X-Origin'] = origin
         return headers
 
+    @staticmethod
+    def _extract_alerts(data):
+        for alert_dict in try_get(data, lambda x: x['alerts'], list) or []:
+            if not isinstance(alert_dict, dict):
+                continue
+            for alert in alert_dict.values():
+                alert_type = alert.get('type')
+                if not alert_type:
+                    continue
+                message = try_get(alert, lambda x: x['text']['simpleText'], compat_str) or ''
+                if message:
+                    yield alert_type, message
+                for run in try_get(alert, lambda x: x['text']['runs'], list) or []:
+                    message += try_get(run, lambda x: x['text'], compat_str)
+                if message:
+                    yield alert_type, message
+
+    def _report_alerts(self, alerts, expected=True):
+        errors = []
+        warnings = []
+        for alert_type, alert_message in alerts:
+            if alert_type.lower() == 'error':
+                errors.append([alert_type, alert_message])
+            else:
+                warnings.append([alert_type, alert_message])
+
+        for alert_type, alert_message in (warnings + errors[:-1]):
+            self.report_warning('YouTube said: %s - %s' % (alert_type, alert_message))
+        if errors:
+            raise ExtractorError('YouTube said: %s' % errors[-1][1], expected=expected)
+
+    def _extract_and_report_alerts(self, data, *args, **kwargs):
+        return self._report_alerts(self._extract_alerts(data), *args, **kwargs)
+
+    def _extract_response(self, item_id, query, note='Downloading API JSON', headers=None,
+                          ytcfg=None, check_get_keys=None, ep='browse', fatal=True, api_hostname=None,
+                          default_client='WEB'):
+        response = None
+        last_error = None
+        count = -1
+        retries = self.get_param('extractor_retries', 3)
+        if check_get_keys is None:
+            check_get_keys = []
+        while count < retries:
+            count += 1
+            if last_error:
+                self.report_warning('%s. Retrying ...' % last_error)
+            try:
+                response = self._call_api(
+                    ep=ep, fatal=True, headers=headers,
+                    video_id=item_id, query=query,
+                    context=self._extract_context(ytcfg, default_client),
+                    api_key=self._extract_api_key(ytcfg, default_client),
+                    api_hostname=api_hostname, default_client=default_client,
+                    note='%s%s' % (note, ' (retry #%d)' % count if count else ''))
+            except ExtractorError as e:
+                if isinstance(e.cause, compat_HTTPError) and e.cause.code in (500, 503, 404):
+                    # Downloading page may result in intermittent 5xx HTTP error
+                    # Sometimes a 404 is also recieved. See: https://github.com/ytdl-org/youtube-dl/issues/28289
+                    last_error = 'HTTP Error %s' % e.cause.code
+                    if count < retries:
+                        continue
+                if fatal:
+                    raise
+                else:
+                    self.report_warning(error_to_compat_str(e))
+                    return
+
+            else:
+                # Youtube may send alerts if there was an issue with the continuation page
+                try:
+                    self._extract_and_report_alerts(response, expected=False)
+                except ExtractorError as e:
+                    if fatal:
+                        raise
+                    self.report_warning(error_to_compat_str(e))
+                    return
+                if not check_get_keys or dict_get(response, check_get_keys):
+                    break
+                # Youtube sometimes sends incomplete data
+                # See: https://github.com/ytdl-org/youtube-dl/issues/28194
+                last_error = 'Incomplete data received'
+                if count >= retries:
+                    if fatal:
+                        raise ExtractorError(last_error)
+                    else:
+                        self.report_warning(last_error)
+                        return
+        return response
+
     @staticmethod
     def is_music_url(url):
         return re.match(r'https?://music\.youtube\.com/', url) is not None
@@ -667,6 +872,11 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
     }
     _SUBTITLE_FORMATS = ('json3', 'srv1', 'srv2', 'srv3', 'ttml', 'vtt')
 
+    _AGE_GATE_REASONS = (
+        'Sign in to confirm your age',
+        'This video may be inappropriate for some users.',
+        'Sorry, this content is age-restricted.')
+
     _GEO_BYPASS = False
 
     IE_NAME = 'youtube'
@@ -1346,7 +1556,32 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             # multiple subtitles with same lang_code
             'url': 'https://www.youtube.com/watch?v=wsQiKKfKxug',
             'only_matching': True,
+        }, {
+            # Force use android client fallback
+            'url': 'https://www.youtube.com/watch?v=YOelRv7fMxY',
+            'info_dict': {
+                'id': 'YOelRv7fMxY',
+                'title': 'Digging a Secret Tunnel from my Workshop',
+                'ext': '3gp',
+                'upload_date': '20210624',
+                'channel_id': 'UCp68_FLety0O-n9QU6phsgw',
+                'uploader': 'colinfurze',
+                'channel_url': r're:https?://(?:www\.)?youtube\.com/channel/UCp68_FLety0O-n9QU6phsgw',
+                'description': 'md5:ecb672623246d98c6c562eed6ae798c3'
+            },
+            'params': {
+                'format': '17',  # 3gp format available on android
+                'extractor_args': {'youtube': {'player_client': ['android']}},
+            },
         },
+        {
+            # Skip download of additional client configs (remix client config in this case)
+            'url': 'https://music.youtube.com/watch?v=MgNrAu2pzNs',
+            'only_matching': True,
+            'params': {
+                'extractor_args': {'youtube': {'player_skip': ['configs']}},
+            },
+        }
     ]
 
     @classmethod
@@ -1364,6 +1599,19 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         self._code_cache = {}
         self._player_cache = {}
 
+    def _extract_player_url(self, ytcfg=None, webpage=None):
+        player_url = try_get(ytcfg, (lambda x: x['PLAYER_JS_URL']), str)
+        if not player_url:
+            player_url = self._search_regex(
+                r'"(?:PLAYER_JS_URL|jsUrl)"\s*:\s*"([^"]+)"',
+                webpage, 'player URL', fatal=False)
+        if player_url.startswith('//'):
+            player_url = 'https:' + player_url
+        elif not re.match(r'https?://', player_url):
+            player_url = compat_urlparse.urljoin(
+                'https://www.youtube.com', player_url)
+        return player_url
+
     def _signature_cache_id(self, example_sig):
         """ Return a string representation of a signature """
         return '.'.join(compat_str(len(part)) for part in example_sig.split('.'))
@@ -1378,6 +1626,15 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             raise ExtractorError('Cannot identify player %r' % player_url)
         return id_m.group('id')
 
+    def _load_player(self, video_id, player_url, fatal=True) -> bool:
+        player_id = self._extract_player_info(player_url)
+        if player_id not in self._code_cache:
+            self._code_cache[player_id] = self._download_webpage(
+                player_url, video_id, fatal=fatal,
+                note='Downloading player ' + player_id,
+                errnote='Download of %s failed' % player_url)
+        return player_id in self._code_cache
+
     def _extract_signature_function(self, video_id, player_url, example_sig):
         player_id = self._extract_player_info(player_url)
 
@@ -1390,20 +1647,16 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         if cache_spec is not None:
             return lambda s: ''.join(s[i] for i in cache_spec)
 
-        if player_id not in self._code_cache:
+        if self._load_player(video_id, player_url):
-            self._code_cache[player_id] = self._download_webpage(
+            code = self._code_cache[player_id]
-                player_url, video_id,
+            res = self._parse_sig_js(code)
-                note='Downloading player ' + player_id,
-                errnote='Download of %s failed' % player_url)
-        code = self._code_cache[player_id]
-        res = self._parse_sig_js(code)
 
-        test_string = ''.join(map(compat_chr, range(len(example_sig))))
+            test_string = ''.join(map(compat_chr, range(len(example_sig))))
-        cache_res = res(test_string)
+            cache_res = res(test_string)
-        cache_spec = [ord(c) for c in cache_res]
+            cache_spec = [ord(c) for c in cache_res]
 
-        self._downloader.cache.store('youtube-sigfuncs', func_id, cache_spec)
+            self._downloader.cache.store('youtube-sigfuncs', func_id, cache_spec)
-        return res
+            return res
 
     def _print_sig_code(self, func, example_sig):
         def gen_sig_code(idxs):
@@ -1474,11 +1727,6 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         if player_url is None:
             raise ExtractorError('Cannot decrypt signature without player_url')
 
-        if player_url.startswith('//'):
-            player_url = 'https:' + player_url
-        elif not re.match(r'https?://', player_url):
-            player_url = compat_urlparse.urljoin(
-                'https://www.youtube.com', player_url)
         try:
             player_id = (player_url, self._signature_cache_id(s))
             if player_id not in self._player_cache:
@@ -1495,6 +1743,31 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             raise ExtractorError(
                 'Signature extraction failed: ' + tb, cause=e)
 
+    def _extract_signature_timestamp(self, video_id, player_url, ytcfg=None, fatal=False):
+        """
+        Extract signatureTimestamp (sts)
+        Required to tell API what sig/player version is in use.
+        """
+        sts = None
+        if isinstance(ytcfg, dict):
+            sts = int_or_none(ytcfg.get('STS'))
+
+        if not sts:
+            # Attempt to extract from player
+            if player_url is None:
+                error_msg = 'Cannot extract signature timestamp without player_url.'
+                if fatal:
+                    raise ExtractorError(error_msg)
+                self.report_warning(error_msg)
+                return
+            if self._load_player(video_id, player_url, fatal=fatal):
+                player_id = self._extract_player_info(player_url)
+                code = self._code_cache[player_id]
+                sts = int_or_none(self._search_regex(
+                    r'(?:signatureTimestamp|sts)\s*:\s*(?P<sts>[0-9]{5})', code,
+                    'JS player signature timestamp', group='sts', fatal=fatal))
+        return sts
+
     def _mark_watched(self, video_id, player_response):
         playback_url = url_or_none(try_get(
             player_response,
@@ -1731,6 +2004,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                         'pbj': 1,
                         'type': 'next',
                     }
+                    if 'itct' in continuation:
+                        query['itct'] = continuation['itct']
                     if parent:
                         query['action_get_comment_replies'] = 1
                     else:
@@ -1776,19 +2051,27 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
 
                     response = try_get(browse,
                                        (lambda x: x['response'],
-                                        lambda x: x[1]['response'])) or {}
+                                        lambda x: x[1]['response']), dict) or {}
 
                     if response.get('continuationContents'):
                         break
 
                     # YouTube sometimes gives reload: now json if something went wrong (e.g. bad auth)
-                    if browse.get('reload'):
+                    if isinstance(browse, dict):
-                        raise ExtractorError('Invalid or missing params in continuation request', expected=False)
+                        if browse.get('reload'):
+                            raise ExtractorError('Invalid or missing params in continuation request', expected=False)
 
-                    # TODO: not tested, merged from old extractor
+                        # TODO: not tested, merged from old extractor
-                    err_msg = browse.get('externalErrorMessage')
+                        err_msg = browse.get('externalErrorMessage')
+                        if err_msg:
+                            last_error = err_msg
+                            continue
+
+                    response_error = try_get(response, lambda x: x['responseContext']['errors']['error'][0], dict) or {}
+                    err_msg = response_error.get('externalErrorMessage')
                     if err_msg:
-                        raise ExtractorError('YouTube said: %s' % err_msg, expected=False)
+                        last_error = err_msg
+                        continue
 
                     # Youtube sometimes sends incomplete data
                     # See: https://github.com/ytdl-org/youtube-dl/issues/28194
@@ -1883,6 +2166,19 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             'comment_count': len(comments),
         }
 
+    @staticmethod
+    def _generate_player_context(sts=None):
+        context = {
+            'html5Preference': 'HTML5_PREF_WANTS',
+        }
+        if sts is not None:
+            context['signatureTimestamp'] = sts
+        return {
+            'playbackContext': {
+                'contentPlaybackContext': context
+            }
+        }
+
     @staticmethod
     def _get_video_info_params(video_id):
         return {
@@ -1904,6 +2200,19 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         webpage = self._download_webpage(
             webpage_url + '&bpctr=9999999999&has_verified=1', video_id, fatal=False)
 
+        ytcfg = self._extract_ytcfg(video_id, webpage) or self._get_default_ytcfg()
+        identity_token = self._extract_identity_token(webpage, video_id)
+        syncid = self._extract_account_syncid(ytcfg)
+        headers = self._generate_api_headers(ytcfg, identity_token, syncid)
+
+        player_url = self._extract_player_url(ytcfg, webpage)
+
+        player_client = try_get(self._configuration_arg('player_client'), lambda x: x[0], str) or ''
+        if player_client.upper() not in ('WEB', 'ANDROID'):
+            player_client = 'WEB'
+        force_mobile_client = player_client.upper() == 'ANDROID'
+        player_skip = self._configuration_arg('player_skip') or []
+
         def get_text(x):
             if not x:
                 return
@@ -1917,37 +2226,68 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
 
         ytm_streaming_data = {}
         if is_music_url:
-            # we are forcing to use parse_json because 141 only appeared in get_video_info.
+            ytm_webpage = None
-            # el, c, cver, cplayer field required for 141(aac 256kbps) codec
+            sts = self._extract_signature_timestamp(video_id, player_url, ytcfg, fatal=False)
-            # maybe paramter of youtube music player?
+            if sts and not force_mobile_client and 'configs' not in player_skip:
-            ytm_player_response = self._parse_json(try_get(compat_parse_qs(
+                ytm_webpage = self._download_webpage(
-                self._download_webpage(
+                    'https://music.youtube.com',
-                    base_url + 'get_video_info', video_id,
+                    video_id, fatal=False, note="Downloading remix client config")
-                    'Fetching youtube music info webpage',
-                    'unable to download youtube music info webpage', query={
-                        **self._get_video_info_params(video_id),
-                        'el': 'detailpage',
-                        'c': 'WEB_REMIX',
-                        'cver': '0.1',
-                        'cplayer': 'UNIPLAYER',
-                    }, fatal=False) or ''),
-                lambda x: x['player_response'][0],
-                compat_str) or '{}', video_id, fatal=False)
-            ytm_streaming_data = ytm_player_response.get('streamingData') or {}
 
+            ytm_cfg = self._extract_ytcfg(video_id, ytm_webpage) or {}
+            ytm_client = 'WEB_REMIX'
+            if not sts or force_mobile_client:
+                # Android client already has signature descrambled
+                # See: https://github.com/TeamNewPipe/NewPipeExtractor/issues/562
+                if not sts:
+                    self.report_warning('Falling back to mobile remix client for player API.')
+                ytm_client = 'ANDROID_MUSIC'
+                ytm_cfg = {}
+
+            ytm_headers = self._generate_api_headers(
+                ytm_cfg, identity_token, syncid,
+                client=ytm_client)
+            ytm_query = {'videoId': video_id}
+            ytm_query.update(self._generate_player_context(sts))
+
+            ytm_player_response = self._extract_response(
+                item_id=video_id, ep='player', query=ytm_query,
+                ytcfg=ytm_cfg, headers=ytm_headers, fatal=False,
+                default_client=ytm_client,
+                note='Downloading %sremix player API JSON' % ('mobile ' if force_mobile_client else ''))
+
+            ytm_streaming_data = try_get(ytm_player_response, lambda x: x['streamingData']) or {}
         player_response = None
         if webpage:
             player_response = self._extract_yt_initial_variable(
                 webpage, self._YT_INITIAL_PLAYER_RESPONSE_RE,
                 video_id, 'initial player response')
 
-        ytcfg = self._extract_ytcfg(video_id, webpage)
+        if not player_response or force_mobile_client:
-        if not player_response:
+            sts = self._extract_signature_timestamp(video_id, player_url, ytcfg, fatal=False)
-            player_response = self._call_api(
+            yt_client = 'WEB'
-                'player', {'videoId': video_id}, video_id, api_key=self._extract_api_key(ytcfg))
+            ytpcfg = ytcfg
+            ytp_headers = headers
+            if not sts or force_mobile_client:
+                # Android client already has signature descrambled
+                # See: https://github.com/TeamNewPipe/NewPipeExtractor/issues/562
+                if not sts:
+                    self.report_warning('Falling back to mobile client for player API.')
+                yt_client = 'ANDROID'
+                ytpcfg = {}
+                ytp_headers = self._generate_api_headers(ytpcfg, identity_token, syncid, yt_client)
 
+            yt_query = {'videoId': video_id}
+            yt_query.update(self._generate_player_context(sts))
+            player_response = self._extract_response(
+                item_id=video_id, ep='player', query=yt_query,
+                ytcfg=ytpcfg, headers=ytp_headers, fatal=False,
+                default_client=yt_client,
+                note='Downloading %splayer API JSON' % ('mobile ' if force_mobile_client else '')
+            )
+
+        # Age-gate workarounds
         playability_status = player_response.get('playabilityStatus') or {}
-        if playability_status.get('reason') == 'Sign in to confirm your age':
+        if playability_status.get('reason') in self._AGE_GATE_REASONS:
             pr = self._parse_json(try_get(compat_parse_qs(
                 self._download_webpage(
                     base_url + 'get_video_info', video_id,
@@ -1955,6 +2295,43 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                     query=self._get_video_info_params(video_id), fatal=False)),
                 lambda x: x['player_response'][0],
                 compat_str) or '{}', video_id)
+            if not pr:
+                self.report_warning('Falling back to embedded-only age-gate workaround.')
+                embed_webpage = None
+                sts = self._extract_signature_timestamp(video_id, player_url, ytcfg, fatal=False)
+                if sts and not force_mobile_client and 'configs' not in player_skip:
+                    embed_webpage = self._download_webpage(
+                        'https://www.youtube.com/embed/%s?html5=1' % video_id,
+                        video_id=video_id, note='Downloading age-gated embed config')
+
+                ytcfg_age = self._extract_ytcfg(video_id, embed_webpage) or {}
+                # If we extracted the embed webpage, it'll tell us if we can view the video
+                embedded_pr = self._parse_json(
+                    try_get(ytcfg_age, lambda x: x['PLAYER_VARS']['embedded_player_response'], str) or '{}',
+                    video_id=video_id)
+                embedded_ps_reason = try_get(embedded_pr, lambda x: x['playabilityStatus']['reason'], str) or ''
+                if embedded_ps_reason not in self._AGE_GATE_REASONS:
+                    yt_client = 'WEB_EMBEDDED_PLAYER'
+                    if not sts or force_mobile_client:
+                        # Android client already has signature descrambled
+                        # See: https://github.com/TeamNewPipe/NewPipeExtractor/issues/562
+                        if not sts:
+                            self.report_warning(
+                                'Falling back to mobile embedded client for player API (note: some formats may be missing).')
+                        yt_client = 'ANDROID_EMBEDDED_PLAYER'
+                        ytcfg_age = {}
+
+                    ytage_headers = self._generate_api_headers(
+                        ytcfg_age, identity_token, syncid, client=yt_client)
+                    yt_age_query = {'videoId': video_id}
+                    yt_age_query.update(self._generate_player_context(sts))
+                    pr = self._extract_response(
+                        item_id=video_id, ep='player', query=yt_age_query,
+                        ytcfg=ytcfg_age, headers=ytage_headers, fatal=False,
+                        default_client=yt_client,
+                        note='Downloading %sage-gated player API JSON' % ('mobile ' if force_mobile_client else '')
+                    ) or {}
+
             if pr:
                 player_response = pr
 
@@ -2026,7 +2403,6 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
 
         formats, itags, stream_ids = [], [], []
         itag_qualities = {}
-        player_url = None
         q = qualities([
             'tiny', 'audio_quality_low', 'audio_quality_medium', 'audio_quality_high',  # Audio only formats
             'small', 'medium', 'large', 'hd720', 'hd1080', 'hd1440', 'hd2160', 'hd2880', 'highres'
@@ -2066,12 +2442,6 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 encrypted_sig = try_get(sc, lambda x: x['s'][0])
                 if not (sc and fmt_url and encrypted_sig):
                     continue
-                if not player_url:
-                    if not webpage:
-                        continue
-                    player_url = self._search_regex(
-                        r'"(?:PLAYER_JS_URL|jsUrl)"\s*:\s*"([^"]+)"',
-                        webpage, 'player URL', fatal=False)
                 if not player_url:
                     continue
                 signature = self._decrypt_signature(sc['s'][0], video_id, player_url)
@@ -2119,8 +2489,12 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                     dct['container'] = dct['ext'] + '_dash'
             formats.append(dct)
 
+        skip_manifests = self._configuration_arg('skip') or []
+        get_dash = 'dash' not in skip_manifests and self.get_param('youtube_include_dash_manifest', True)
+        get_hls = 'hls' not in skip_manifests and self.get_param('youtube_include_hls_manifest', True)
+
         for sd in (streaming_data, ytm_streaming_data):
-            hls_manifest_url = sd.get('hlsManifestUrl')
+            hls_manifest_url = get_hls and sd.get('hlsManifestUrl')
             if hls_manifest_url:
                 for f in self._extract_m3u8_formats(
                         hls_manifest_url, video_id, 'mp4', fatal=False):
@@ -2130,23 +2504,21 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                         f['format_id'] = itag
                     formats.append(f)
 
-        if self.get_param('youtube_include_dash_manifest', True):
+            dash_manifest_url = get_dash and sd.get('dashManifestUrl')
-            for sd in (streaming_data, ytm_streaming_data):
+            if dash_manifest_url:
-                dash_manifest_url = sd.get('dashManifestUrl')
+                for f in self._extract_mpd_formats(
-                if dash_manifest_url:
+                        dash_manifest_url, video_id, fatal=False):
-                    for f in self._extract_mpd_formats(
+                    itag = f['format_id']
-                            dash_manifest_url, video_id, fatal=False):
+                    if itag in itags:
-                        itag = f['format_id']
+                        continue
-                        if itag in itags:
+                    if itag in itag_qualities:
-                            continue
+                        f['quality'] = q(itag_qualities[itag])
-                        if itag in itag_qualities:
+                    filesize = int_or_none(self._search_regex(
-                            f['quality'] = q(itag_qualities[itag])
+                        r'/clen/(\d+)', f.get('fragment_base_url')
-                        filesize = int_or_none(self._search_regex(
+                        or f['url'], 'file size', default=None))
-                            r'/clen/(\d+)', f.get('fragment_base_url')
+                    if filesize:
-                            or f['url'], 'file size', default=None))
+                        f['filesize'] = filesize
-                        if filesize:
+                    formats.append(f)
-                            f['filesize'] = filesize
-                        formats.append(f)
 
         if not formats:
             if not self.get_param('allow_unplayable_formats') and streaming_data.get('licenseInfos'):
@@ -2232,6 +2604,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             or microformat.get('lengthSeconds')) \
             or parse_duration(search_meta('duration'))
         is_live = video_details.get('isLive')
+        is_upcoming = video_details.get('isUpcoming')
         owner_profile_url = microformat.get('ownerProfileUrl')
 
         info = {
@@ -2305,7 +2678,9 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                         continue
                     process_language(
                         automatic_captions, base_url, translation_language_code,
-                        try_get(translation_language, lambda x: x['languageName']['simpleText']),
+                        try_get(translation_language, (
+                            lambda x: x['languageName']['simpleText'],
+                            lambda x: x['languageName']['runs'][0]['text'])),
                         {'tlang': translation_language_code})
                 info['automatic_captions'] = automatic_captions
         info['subtitles'] = subtitles
@@ -2343,8 +2718,10 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 webpage, self._YT_INITIAL_DATA_RE, video_id,
                 'yt initial data')
         if not initial_data:
-            initial_data = self._call_api(
+            initial_data = self._extract_response(
-                'next', {'videoId': video_id}, video_id, fatal=False, api_key=self._extract_api_key(ytcfg))
+                item_id=video_id, ep='next', fatal=False,
+                ytcfg=ytcfg, headers=headers, query={'videoId': video_id},
+                note='Downloading initial data API JSON')
 
         try:
             # This will error if there is no livechat
@@ -2353,7 +2730,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 'url': 'https://www.youtube.com/watch?v=%s' % video_id,  # url is needed to set cookies
                 'video_id': video_id,
                 'ext': 'json',
-                'protocol': 'youtube_live_chat' if is_live else 'youtube_live_chat_replay',
+                'protocol': 'youtube_live_chat' if is_live or is_upcoming else 'youtube_live_chat_replay',
             }]
         except (KeyError, IndexError, TypeError):
             pass
@@ -3500,40 +3877,6 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             self._extract_mix_playlist(playlist, playlist_id, data, webpage),
             playlist_id=playlist_id, playlist_title=title)
 
-    @staticmethod
-    def _extract_alerts(data):
-        for alert_dict in try_get(data, lambda x: x['alerts'], list) or []:
-            if not isinstance(alert_dict, dict):
-                continue
-            for alert in alert_dict.values():
-                alert_type = alert.get('type')
-                if not alert_type:
-                    continue
-                message = try_get(alert, lambda x: x['text']['simpleText'], compat_str) or ''
-                if message:
-                    yield alert_type, message
-                for run in try_get(alert, lambda x: x['text']['runs'], list) or []:
-                    message += try_get(run, lambda x: x['text'], compat_str)
-                if message:
-                    yield alert_type, message
-
-    def _report_alerts(self, alerts, expected=True):
-        errors = []
-        warnings = []
-        for alert_type, alert_message in alerts:
-            if alert_type.lower() == 'error':
-                errors.append([alert_type, alert_message])
-            else:
-                warnings.append([alert_type, alert_message])
-
-        for alert_type, alert_message in (warnings + errors[:-1]):
-            self.report_warning('YouTube said: %s - %s' % (alert_type, alert_message))
-        if errors:
-            raise ExtractorError('YouTube said: %s' % errors[-1][1], expected=expected)
-
-    def _extract_and_report_alerts(self, data, *args, **kwargs):
-        return self._report_alerts(self._extract_alerts(data), *args, **kwargs)
-
     def _reload_with_unavailable_videos(self, item_id, data, webpage):
         """
         Get playlist with unavailable videos if the 'show unavailable videos' button exists.
@@ -3578,60 +3921,6 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                 check_get_keys='contents', fatal=False,
                 note='Downloading API JSON with unavailable videos')
 
-    def _extract_response(self, item_id, query, note='Downloading API JSON', headers=None,
-                          ytcfg=None, check_get_keys=None, ep='browse', fatal=True):
-        response = None
-        last_error = None
-        count = -1
-        retries = self.get_param('extractor_retries', 3)
-        if check_get_keys is None:
-            check_get_keys = []
-        while count < retries:
-            count += 1
-            if last_error:
-                self.report_warning('%s. Retrying ...' % last_error)
-            try:
-                response = self._call_api(
-                    ep=ep, fatal=True, headers=headers,
-                    video_id=item_id, query=query,
-                    context=self._extract_context(ytcfg),
-                    api_key=self._extract_api_key(ytcfg),
-                    note='%s%s' % (note, ' (retry #%d)' % count if count else ''))
-            except ExtractorError as e:
-                if isinstance(e.cause, compat_HTTPError) and e.cause.code in (500, 503, 404):
-                    # Downloading page may result in intermittent 5xx HTTP error
-                    # Sometimes a 404 is also recieved. See: https://github.com/ytdl-org/youtube-dl/issues/28289
-                    last_error = 'HTTP Error %s' % e.cause.code
-                    if count < retries:
-                        continue
-                if fatal:
-                    raise
-                else:
-                    self.report_warning(error_to_compat_str(e))
-                    return
-
-            else:
-                # Youtube may send alerts if there was an issue with the continuation page
-                try:
-                    self._extract_and_report_alerts(response, expected=False)
-                except ExtractorError as e:
-                    if fatal:
-                        raise
-                    self.report_warning(error_to_compat_str(e))
-                    return
-                if not check_get_keys or dict_get(response, check_get_keys):
-                    break
-                # Youtube sometimes sends incomplete data
-                # See: https://github.com/ytdl-org/youtube-dl/issues/28194
-                last_error = 'Incomplete data received'
-                if count >= retries:
-                    if fatal:
-                        raise ExtractorError(last_error)
-                    else:
-                        self.report_warning(last_error)
-                        return
-        return response
-
     def _extract_webpage(self, url, item_id):
         retries = self.get_param('extractor_retries', 3)
         count = -1

yt_dlp/options.py

@@ -716,7 +716,8 @@ def parseOpts(overrideArguments=None):
         help=(
             'Give these arguments to the external downloader. '
             'Specify the downloader name and the arguments separated by a colon ":". '
-            'You can use this option multiple times (Alias: --external-downloader-args)'))
+            'You can use this option multiple times to give different arguments to different downloaders '
+            '(Alias: --external-downloader-args)'))
 
     workarounds = optparse.OptionGroup(parser, 'Workarounds')
     workarounds.add_option(
@@ -1343,22 +1344,34 @@ def parseOpts(overrideArguments=None):
         '--no-hls-split-discontinuity',
         dest='hls_split_discontinuity', action='store_false',
         help='Do not split HLS playlists to different formats at discontinuities such as ad breaks (default)')
+    extractor.add_option(
+        '--extractor-args',
+        metavar='KEY:ARGS', dest='extractor_args', default={}, type='str',
+        action='callback', callback=_dict_from_options_callback,
+        callback_kwargs={
+            'multiple_keys': False,
+            'process': lambda val: dict(
+                (lambda x: (x[0], x[1].split(',')))(arg.split('=', 1) + ['', '']) for arg in val.split(';'))
+        },
+        help=(
+            'Pass these arguments to the extractor. See "EXTRACTOR ARGUMENTS" for details. '
+            'You can use this option multiple times to give different arguments to different extractors'))
     extractor.add_option(
         '--youtube-include-dash-manifest', '--no-youtube-skip-dash-manifest',
         action='store_true', dest='youtube_include_dash_manifest', default=True,
-        help='Download the DASH manifests and related data on YouTube videos (default) (Alias: --no-youtube-skip-dash-manifest)')
+        help=optparse.SUPPRESS_HELP)
     extractor.add_option(
         '--youtube-skip-dash-manifest', '--no-youtube-include-dash-manifest',
         action='store_false', dest='youtube_include_dash_manifest',
-        help='Do not download the DASH manifests and related data on YouTube videos (Alias: --no-youtube-include-dash-manifest)')
+        help=optparse.SUPPRESS_HELP)
     extractor.add_option(
         '--youtube-include-hls-manifest', '--no-youtube-skip-hls-manifest',
         action='store_true', dest='youtube_include_hls_manifest', default=True,
-        help='Download the HLS manifests and related data on YouTube videos (default) (Alias: --no-youtube-skip-hls-manifest)')
+        help=optparse.SUPPRESS_HELP)
     extractor.add_option(
         '--youtube-skip-hls-manifest', '--no-youtube-include-hls-manifest',
         action='store_false', dest='youtube_include_hls_manifest',
-        help='Do not download the HLS manifests and related data on YouTube videos (Alias: --no-youtube-include-hls-manifest)')
+        help=optparse.SUPPRESS_HELP)
 
     parser.add_option_group(general)
     parser.add_option_group(network)

yt_dlp/postprocessor/embedthumbnail.py

@@ -92,7 +92,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
         # format, there will be some additional data loss.
         # PNG, on the other hand, is lossless.
         thumbnail_ext = os.path.splitext(thumbnail_filename)[1][1:]
-        if thumbnail_ext not in ('jpg', 'png'):
+        if thumbnail_ext not in ('jpg', 'jpeg', 'png'):
             thumbnail_filename = convertor.convert_thumbnail(thumbnail_filename, 'png')
             thumbnail_ext = 'png'
 

yt_dlp/postprocessor/ffmpeg.py

@@ -896,6 +896,8 @@ class FFmpegThumbnailsConvertorPP(FFmpegPostProcessor):
             _, thumbnail_ext = os.path.splitext(original_thumbnail)
             if thumbnail_ext:
                 thumbnail_ext = thumbnail_ext[1:].lower()
+            if thumbnail_ext == 'jpeg':
+                thumbnail_ext = 'jpg'
             if thumbnail_ext == self.format:
                 self.to_screen('Thumbnail "%s" is already in the requested format' % original_thumbnail)
                 continue

yt_dlp/utils.py

@@ -3976,20 +3976,23 @@ class LazyList(collections.Sequence):
     def __iter__(self):
         if self.__reversed:
             # We need to consume the entire iterable to iterate in reverse
-            yield from self.exhaust()[::-1]
+            yield from self.exhaust()
             return
         yield from self.__cache
         for item in self.__iterable:
             self.__cache.append(item)
             yield item
 
-    def exhaust(self):
+    def __exhaust(self):
-        ''' Evaluate the entire iterable '''
         self.__cache.extend(self.__iterable)
         return self.__cache
 
+    def exhaust(self):
+        ''' Evaluate the entire iterable '''
+        return self.__exhaust()[::-1 if self.__reversed else 1]
+
     @staticmethod
-    def _reverse_index(x):
+    def __reverse_index(x):
         return -(x + 1)
 
     def __getitem__(self, idx):
@@ -3998,18 +4001,18 @@ class LazyList(collections.Sequence):
             start = idx.start if idx.start is not None else 0 if step > 0 else -1
             stop = idx.stop if idx.stop is not None else -1 if step > 0 else 0
             if self.__reversed:
-                start, stop, step = map(self._reverse_index, (start, stop, step))
+                (start, stop), step = map(self.__reverse_index, (start, stop)), -step
                 idx = slice(start, stop, step)
         elif isinstance(idx, int):
             if self.__reversed:
-                idx = self._reverse_index(idx)
+                idx = self.__reverse_index(idx)
             start = stop = idx
         else:
             raise TypeError('indices must be integers or slices')
         if start < 0 or stop < 0:
             # We need to consume the entire iterable to be able to slice from the end
             # Obviously, never use this with infinite iterables
-            return self.exhaust()[idx]
+            return self.__exhaust()[idx]
 
         n = max(start, stop) - len(self.__cache) + 1
         if n > 0:
@@ -4027,7 +4030,7 @@ class LazyList(collections.Sequence):
         self.exhaust()
         return len(self.__cache)
 
-    def __reversed__(self):
+    def reverse(self):
         self.__reversed = not self.__reversed
         return self
 

yt_dlp/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '2021.06.09'
+__version__ = '2021.06.23'