Release 2021.02.15

[youtube] Fix for new accounts
Cookies for some new accounts doesn't work with age-gated videos without `has_verified=1`
2021-02-16 04:04:27 +05:30 · 2021-02-16 03:20:06 +05:30 · 2021-02-16 02:36:42 +05:30 · 2021-02-16 00:46:23 +05:30 · 2021-02-15 23:22:11 +05:30 · 2021-02-15 20:08:27 +05:30
68 changed files with 1523 additions and 838 deletions
@@ -21,7 +21,7 @@ assignees: ''
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 - [ ] I'm reporting a broken site support
- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.01.29
+ [debug] yt-dlp version 2021.02.09
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
@@ -21,7 +21,7 @@ assignees: ''
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 - [ ] I'm reporting a new site support request
- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones
@@ -21,13 +21,13 @@ assignees: ''
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 - [ ] I'm reporting a site feature request
- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
@@ -21,7 +21,7 @@ assignees: ''
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 - [ ] I'm reporting a broken site support issue
- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.01.29
+ [debug] yt-dlp version 2021.02.09
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
@@ -21,13 +21,13 @@ assignees: ''
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 - [ ] I'm reporting a feature request
- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
@@ -55,9 +55,7 @@ jobs:
        asset_content_type: application/octet-stream
    - name: Get SHA2-256SUMS for youtube-dlc
      id: sha2_file
-      env:
+      run: echo "::set-output name=sha2_unix::$(sha256sum youtube-dlc)"
        SHA2: ${{ hashFiles('youtube-dlc') }}
      run: echo "::set-output name=sha2_unix::$SHA2"
    - name: Install dependencies for pypi
      run: |
        python -m pip install --upgrade pip
@@ -75,6 +73,9 @@ jobs:
    runs-on: windows-latest
    outputs:
      sha2_windows: ${{ steps.sha2_file_win.outputs.sha2_windows }}
    needs: build_unix
    steps:
@@ -84,14 +85,14 @@ jobs:
      with:
          python-version: '3.8'
    - name: Install Requirements
-      run: pip install pyinstaller mutagen
+      run: pip install pyinstaller mutagen pycryptodome
    - name: Bump version
      id: bump_version
      run: python devscripts/update-version.py
    - name: Print version
      run: echo "${{ steps.bump_version.outputs.ytdlc_version }}"
    - name: Run PyInstaller Script
-      run: python devscripts/pyinst.py 64
+      run: python pyinst.py 64
    - name: Upload youtube-dlc.exe Windows binary
      id: upload-release-windows
      uses: actions/upload-release-asset@v1
@@ -104,14 +105,15 @@ jobs:
        asset_content_type: application/vnd.microsoft.portable-executable
    - name: Get SHA2-256SUMS for youtube-dlc.exe
      id: sha2_file_win
-      env:
+      run: echo "::set-output name=sha2_windows::$(certUtil -hashfile dist\youtube-dlc.exe SHA256 | findstr -v :)"
        SHA2_win: ${{ hashFiles('dist/youtube-dlc.exe') }}
      run: echo "::set-output name=sha2_windows::$SHA2_win"
  build_windows32:
    runs-on: windows-latest
    outputs:
      sha2_windows32: ${{ steps.sha2_file_win32.outputs.sha2_windows32 }}
    needs: [build_unix, build_windows]
    steps:
@@ -121,15 +123,17 @@ jobs:
      with:
          python-version: '3.4.4'
          architecture: 'x86'
    - name: Install VS libs
      run: choco install vcexpress2010
    - name: Install Requirements for 32 Bit
-      run: pip install pyinstaller==3.5 mutagen
+      run: pip install pyinstaller==3.5 mutagen pycryptodome
    - name: Bump version
      id: bump_version
      run: python devscripts/update-version.py
    - name: Print version
      run: echo "${{ steps.bump_version.outputs.ytdlc_version }}"
    - name: Run PyInstaller Script for 32 Bit
-      run: python devscripts/pyinst.py 32
+      run: python pyinst.py 32
    - name: Upload Executable youtube-dlc_x86.exe
      id: upload-release-windows32
      uses: actions/upload-release-asset@v1
@@ -142,9 +146,7 @@ jobs:
        asset_content_type: application/vnd.microsoft.portable-executable
    - name: Get SHA2-256SUMS for youtube-dlc_x86.exe
      id: sha2_file_win32
-      env:
+      run: echo "::set-output name=sha2_windows32::$(certUtil -hashfile dist\youtube-dlc_x86.exe SHA256 | findstr -v :)"
        SHA2_win32: ${{ hashFiles('dist/youtube-dlc_x86.exe') }}
      run: echo "::set-output name=sha2_windows32::$SHA2_win32"
    - name: Make SHA2-256SUMS file
      env:
        SHA2_WINDOWS: ${{ needs.build_windows.outputs.sha2_windows }}
@@ -17,6 +17,7 @@ MANIFEST
 test/local_parameters.json
 .coverage
 cover/
 secrets/
 updates_key.pem
 *.egg-info
 .tox
@@ -54,10 +55,17 @@ youtube-dlc
 *.swf
 *.part
 *.ytdl
 *.frag
 *.frag.urls
 *.aria2
 *.swp
 *.ogg
 *.opus
 *.info.json
 *.live_chat.json
 *.jpg
 *.png
 *.webp
 *.annotations.xml
 *.description
@@ -17,3 +17,4 @@ alxnull
 FelixFrog
 Zocker1999NET
 nao20010128nao
 shirt-dev
@@ -17,12 +17,56 @@
 -->
 ### 2021.02.15
 * **Merge youtube-dl:** Upto [2021.02.10](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.10) (except archive.org)
 * [niconico] Improved extraction and support encrypted/SMILE movies
 * Fix HLS AES-128 with multiple keys in external downloaders
 * [youtube_live_chat] Fix by using POST API
 * [rumble] Add support for video page
 * Option to allow downloading unplayable video formats (`--allow-unplayable-formats`)
 * [ExtractAudio] Don't re-encode when file is already in a common audio format
 * Change optional dependency to `pycryptodome`
 * [youtube] Fix search continuations
 * [youtube] Fix for new accounts
 * Improve build/updater:
    * Fix SHA256 calculation in build and implement hash checking for updater
    * Exit immediately in windows once the update process starts
    * Fix updater for `x86.exe`
    * Updater looks for both `yt-dlp` and `youtube-dlc` in releases for future-proofing
 * Fix issue with unicode filenames in aria2c
 * Fix `allow_playlist_files` not being correctly passed through
 * Fix for empty HTTP head requests
 * Fix `get_executable_path` in UNIX
 * [sponskrub] Print ffmpeg output and errors to terminal
 * `__real_download` should be false when ffmpeg unavailable and no download
 * Show `exe`/`zip`/`source` and 32/64bit in verbose message
 ### 2021.02.09
 * **aria2c support for DASH/HLS**: by [shirt](https://github.com/shirt-dev)
 * **Implement Updater** (`-U`) by [shirt](https://github.com/shirt-dev)
 * [youtube] Fix comment extraction
 * [youtube_live_chat] Improve extraction
 * [youtube] Fix for channel URLs sometimes not downloading all pages
 * [aria2c] Changed default arguments to `--console-log-level=warn --summary-interval=0 --file-allocation=none -x16 -j16 -s16`
 * Add fallback for thumbnails
 * [embedthumbnail] Keep original thumbnail after conversion if write_thumbnail given
 * [embedsubtitle] Keep original subtitle after conversion if write_subtitles given
 * [pyinst.py] Move back to root dir
 * [youtube] Simplified renderer parsing and bugfixes
 * [movefiles] Fix compatibility with python2
 * [remuxvideo] Fix validation of conditional remux
 * [sponskrub] Don't raise error when the video does not exist
 * [documentation] Crypto is an optional dependency
 ### 2021.02.04
 * **Merge youtube-dl:** Upto [2021.02.04.1](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.04.1)
-* **Date/time formatting in output template:** You can now use [`strftime`](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) to format date/time fields. Example: `%(upload_date>%Y-%m-%d)s`
+* **Date/time formatting in output template:**
    * You can use [`strftime`](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) to format date/time fields. Example: `%(upload_date>%Y-%m-%d)s`
 * **Multiple output templates:**
-    * Seperate output templates can be given for the different metadata files by using `-o TYPE:TEMPLATE`
+    * Separate output templates can be given for the different metadata files by using `-o TYPE:TEMPLATE`
-    * The alowed types are: `subtitle|thumbnail|description|annotation|infojson|pl_description|pl_infojson`
+    * The allowed types are: `subtitle|thumbnail|description|annotation|infojson|pl_description|pl_infojson`
 * [youtube] More metadata extraction for channel/playlist URLs (channel, uploader, thumbnail, tags)
 * New option `--no-write-playlist-metafiles` to prevent writing playlist metadata files
 * [audius] Fix extractor
@@ -38,7 +82,7 @@
 ### 2021.01.29
-* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: Co-authored by @animelover1984 and @bbepis
+* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: Co-authored by [animelover1984](https://github.com/animelover1984) and [bbepis](https://github.com/bbepis)
    * Add `--get-comments`
    * [youtube] Extract comments
    * [billibilli] Added BiliBiliSearchIE, BilibiliChannelIE
@@ -78,7 +122,7 @@
    * Valid types are: home, temp, description, annotation, subtitle, infojson, thumbnail
    * Additionally, configuration file is taken from home directory or current directory ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=Home%20Configuration))
 * Allow passing different arguments to different external downloaders ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=--downloader-args%20NAME:ARGS))
-* [mildom] Add extractor by @nao20010128nao
+* [mildom] Add extractor by [nao20010128nao](https://github.com/nao20010128nao)
 * Warn when using old style `--external-downloader-args` and `--post-processor-args`
 * Fix `--no-overwrite` when using `--write-link`
 * [sponskrub] Output `unrecognized argument` error message correctly
@@ -109,7 +153,7 @@
 ### 2021.01.14
 * Added option `--break-on-reject`
-* [roosterteeth.com] Fix for bonus episodes by @Zocker1999NET
+* [roosterteeth.com] Fix for bonus episodes by [Zocker1999NET](https://github.com/Zocker1999NET)
 * [tiktok] Fix for when share_info is empty
 * [EmbedThumbnail] Fix bug due to incorrect function name
 * [documentation] Changed sponskrub links to point to [pukkandan/sponskrub](https://github.com/pukkandan/SponSkrub) since I am now providing both linux and windows releases
@@ -118,18 +162,18 @@
 ### 2021.01.12
-* [roosterteeth.com] Add subtitle support by @samiksome
+* [roosterteeth.com] Add subtitle support by [samiksome](https://github.com/samiksome)
-* Added `--force-overwrites`, `--no-force-overwrites` by @alxnull
+* Added `--force-overwrites`, `--no-force-overwrites` by [alxnull](https://github.com/alxnull)
 * Changed fork name to `yt-dlp`
-* Fix typos by @FelixFrog
+* Fix typos by [FelixFrog](https://github.com/FelixFrog)
 * [ci] Option to skip
 * [changelog] Added unreleased changes in blackjack4494/yt-dlc
 ### 2021.01.10
-* [archive.org] Fix extractor and add support for audio and playlists by @wporr
+* [archive.org] Fix extractor and add support for audio and playlists by [wporr](https://github.com/wporr)
-* [Animelab] Added by @mariuszskon
+* [Animelab] Added by [mariuszskon](https://github.com/mariuszskon)
-* [youtube:search] Fix view_count by @ohnonot
+* [youtube:search] Fix view_count by [ohnonot](https://github.com/ohnonot)
 * [youtube] Show if video is embeddable in info
 * Update version badge automatically in README
 * Enable `test_youtube_search_matching`
@@ -138,11 +182,11 @@
 ### 2021.01.09
 * [youtube] Fix bug in automatic caption extraction
-* Add `post_hooks` to YoutubeDL by @alexmerkel
+* Add `post_hooks` to YoutubeDL by [alexmerkel](https://github.com/alexmerkel)
-* Batch file enumeration improvements by @glenn-slayden
+* Batch file enumeration improvements by [glenn-slayden](https://github.com/glenn-slayden)
-* Stop immediately when reaching `--max-downloads` by @glenn-slayden
+* Stop immediately when reaching `--max-downloads` by [glenn-slayden](https://github.com/glenn-slayden)
-* Fix incorrect ANSI sequence for restoring console-window title by @glenn-slayden
+* Fix incorrect ANSI sequence for restoring console-window title by [glenn-slayden](https://github.com/glenn-slayden)
-* Kill child processes when yt-dlc is killed by @Unrud
+* Kill child processes when yt-dlc is killed by [Unrud](https://github.com/Unrud)
 ### 2021.01.08
@@ -152,11 +196,11 @@
 ### 2021.01.07-1
-* [Akamai] fix by @nixxo
+* [Akamai] fix by [nixxo](https://github.com/nixxo)
-* [Tiktok] merge youtube-dl tiktok extractor by @GreyAlien502
+* [Tiktok] merge youtube-dl tiktok extractor by [GreyAlien502](https://github.com/GreyAlien502)
-* [vlive] add support for playlists by @kyuyeunk
+* [vlive] add support for playlists by [kyuyeunk](https://github.com/kyuyeunk)
-* [youtube_live_chat] make sure playerOffsetMs is positive by @siikamiika
+* [youtube_live_chat] make sure playerOffsetMs is positive by [siikamiika](https://github.com/siikamiika)
-* Ignore extra data streams in ffmpeg by @jbruchon
+* Ignore extra data streams in ffmpeg by [jbruchon](https://github.com/jbruchon)
 * Allow passing different arguments to different postprocessors using `--postprocessor-args`
 * Deprecated `--sponskrub-args`. The same can now be done using `--postprocessor-args "sponskrub:<args>"`
 * [CI] Split tests into core-test and full-test
@@ -186,15 +230,15 @@
    * Changed video format sorting to show video only files and video+audio files together.
    * Added `--video-multistreams`, `--no-video-multistreams`, `--audio-multistreams`, `--no-audio-multistreams`
    * Added `b`,`w`,`v`,`a` as alias for `best`, `worst`, `video` and `audio` respectively
-* **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by @h-h-h-h - See [Internet Shortcut Options]README.md(#internet-shortcut-options) for details
+* **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by [h-h-h-h](https://github.com/h-h-h-h) - See [Internet Shortcut Options](README.md#internet-shortcut-options) for details
 * **Sponskrub integration:** Added `--sponskrub`, `--sponskrub-cut`, `--sponskrub-force`, `--sponskrub-location`, `--sponskrub-args` - See [SponSkrub Options](README.md#sponskrub-options-sponsorblock) for details
-* Added `--force-download-archive` (`--force-write-archive`) by @h-h-h-h
+* Added `--force-download-archive` (`--force-write-archive`) by [h-h-h-h](https://github.com/h-h-h-h)
 * Added `--list-formats-as-table`,  `--list-formats-old`
 * **Negative Options:** Makes it possible to negate most boolean options by adding a `no-` to the switch. Usefull when you want to reverse an option that is defined in a config file
    * Added `--no-ignore-dynamic-mpd`, `--no-allow-dynamic-mpd`, `--allow-dynamic-mpd`, `--youtube-include-hls-manifest`, `--no-youtube-include-hls-manifest`, `--no-youtube-skip-hls-manifest`, `--no-download`, `--no-download-archive`, `--resize-buffer`, `--part`, `--mtime`, `--no-keep-fragments`, `--no-cookies`, `--no-write-annotations`, `--no-write-info-json`, `--no-write-description`, `--no-write-thumbnail`, `--youtube-include-dash-manifest`, `--post-overwrites`, `--no-keep-video`, `--no-embed-subs`, `--no-embed-thumbnail`, `--no-add-metadata`, `--no-include-ads`, `--no-write-sub`, `--no-write-auto-sub`, `--no-playlist-reverse`, `--no-restrict-filenames`, `--youtube-include-dash-manifest`, `--no-format-sort-force`, `--flat-videos`, `--no-list-formats-as-table`, `--no-sponskrub`, `--no-sponskrub-cut`, `--no-sponskrub-force`
    * Renamed: `--write-subs`, `--no-write-subs`, `--no-write-auto-subs`, `--write-auto-subs`. Note that these can still be used without the ending "s"
 * Relaxed validation for format filters so that any arbitrary field can be used
-* Fix for embedding thumbnail in mp3 by @pauldubois98 ([ytdl-org/youtube-dl#21569](https://github.com/ytdl-org/youtube-dl/pull/21569))
+* Fix for embedding thumbnail in mp3 by [pauldubois98](https://github.com/pauldubois98) ([ytdl-org/youtube-dl#21569](https://github.com/ytdl-org/youtube-dl/pull/21569))
 * Make Twitch Video ID output from Playlist and VOD extractor same. This is only a temporary fix
 * **Merge youtube-dl:** Upto [2021.01.03](https://github.com/ytdl-org/youtube-dl/commit/8e953dcbb10a1a42f4e12e4e132657cb0100a1f8) - See [blackjack4494/yt-dlc#280](https://github.com/blackjack4494/yt-dlc/pull/280) for details
    * Extractors [tiktok](https://github.com/ytdl-org/youtube-dl/commit/fb626c05867deab04425bad0c0b16b55473841a2) and [hotstar](https://github.com/ytdl-org/youtube-dl/commit/bb38a1215718cdf36d73ff0a7830a64cd9fa37cc) have not been merged
@@ -211,7 +255,7 @@
    * Redirect channel home to /video
    * Print youtube's warning message
    * Multiple pages are handled better for feeds
-* Add --break-on-existing by @gergesh
+* Add --break-on-existing by [gergesh](https://github.com/gergesh)
 * Pre-check video IDs in the archive before downloading
 * [bitwave.tv] New extractor
 * [Gedi] Add extractor
@@ -4,7 +4,7 @@ man: README.txt youtube-dlc.1 youtube-dlc.bash-completion youtube-dlc.zsh youtub
 clean:
-	rm -rf youtube-dlc.1.temp.md youtube-dlc.1 youtube-dlc.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dlc.tar.gz youtube-dlc.zsh youtube-dlc.fish youtube_dlc/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.spec CONTRIBUTING.md.tmp youtube-dlc youtube-dlc.exe
+	rm -rf youtube-dlc.1.temp.md youtube-dlc.1 youtube-dlc.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dlc.tar.gz youtube-dlc.zsh youtube-dlc.fish youtube_dlc/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.spec *.frag *.frag.urls *.frag.aria2 CONTRIBUTING.md.tmp youtube-dlc youtube-dlc.exe
 	find . -name "*.pyc" -delete
 	find . -name "*.class" -delete
@@ -2,7 +2,9 @@
 [![Release version](https://img.shields.io/github/v/release/pukkandan/yt-dlp?color=brightgreen&label=Release)](https://github.com/pukkandan/yt-dlp/releases/latest)
 [![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg)](LICENSE)
-[![CI Status](https://github.com/pukkandan/yt-dlp/workflows/Core%20Tests/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlp/actions)  
+[![CI Status](https://github.com/pukkandan/yt-dlp/workflows/Core%20Tests/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlp/actions)
 [![Discord](https://img.shields.io/discord/807245652072857610?color=blue&label=discord&logo=discord)](https://discord.gg/S75JaBna)  
 [![Commits](https://img.shields.io/github/commit-activity/m/pukkandan/yt-dlp?label=commits)](https://github.com/pukkandan/yt-dlp/commits)
 [![Last Commit](https://img.shields.io/github/last-commit/pukkandan/yt-dlp/master)](https://github.com/pukkandan/yt-dlp/commits)
 [![Downloads](https://img.shields.io/github/downloads/pukkandan/yt-dlp/total)](https://github.com/pukkandan/yt-dlp/releases/latest)
@@ -54,18 +56,20 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 * **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
-* **Merged with youtube-dl v2021.02.04.1**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
+* **Merged with youtube-dl v2021.02.10**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
 * **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--get-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, Playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/pukkandan/yt-dlp/pull/31) for details.
 * **Youtube improvements**:
-    * All Youtube Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) works correctly and support downloading multiple pages of content
+    * All Youtube Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) works correctly and supports downloading multiple pages of content
-    * Youtube search works correctly (`ytsearch:`, `ytsearchdate:`) along with Search URLs
+    * Youtube search (`ytsearch:`, `ytsearchdate:`) along with Search URLs works correctly
    * Redirect channel's home URL automatically to `/video` to preserve the old behaviour
 * **Aria2c with HLS/DASH**: You can use aria2c as the external downloader for DASH(mpd) and HLS(m3u8) formats. No more slow ffmpeg/native downloads
 * **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius
-* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, tiktok, akamai, ina
+* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, tiktok, akamai, ina, rumble
 * **Plugin support**: Extractors can be loaded from an external file. See [plugins](#plugins) for details
@@ -79,6 +83,8 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 * **Improvements**: Multiple `--postprocessor-args` and `--external-downloader-args`, Date/time formatting in `-o`, faster archive checking, more [format selection options](#format-selection) etc
 * **Self-updater**: The releases can be updated using `youtube-dlc -U`
 See [changelog](Changelog.md) or [commits](https://github.com/pukkandan/yt-dlp/commits) for the full list of changes
@@ -91,22 +97,23 @@ If you are coming from [youtube-dl](https://github.com/ytdl-org/youtube-dl), the
 # INSTALLATION
 You can install yt-dlp using one of the following methods:
 * Download the binary from the [latest release](https://github.com/pukkandan/yt-dlp/releases/latest) (recommended method)
 * Use [PyPI package](https://pypi.org/project/yt-dlp): `python -m pip install --upgrade yt-dlp`
 * Download the binary from the [latest release](https://github.com/pukkandan/yt-dlp/releases/latest)
 * Use pip+git: `python -m pip install --upgrade git+https://github.com/pukkandan/yt-dlp.git@release`
 * Install master branch: `python -m pip install --upgrade git+https://github.com/pukkandan/yt-dlp`
 ### UPDATE
-`-U` does not work. Simply repeat the install process to update.
+Starting from version `2021.02.09`, you can use `youtube-dlc -U` to update if you are using the provided release.
 If you are using `pip`, simply re-run the same command that was used to install the program.
 ### COMPILE
 **For Windows**:
-To build the Windows executable, you must have pyinstaller (and optionally mutagen for embedding thumbnail in opus/ogg files)
+To build the Windows executable, you must have pyinstaller (and optionally mutagen and pycryptodome)
-    python -m pip install --upgrade pyinstaller mutagen
+    python -m pip install --upgrade pyinstaller mutagen pycryptodome
-Once you have all the necessary dependancies installed, just run `py devscripts\pyinst.py`. The executable will be built for the same architecture (32/64 bit) as the python used to build it. It is strongly reccomended to use python3 although python2.6+ is supported.
+Once you have all the necessary dependancies installed, just run `py pyinst.py`. The executable will be built for the same architecture (32/64 bit) as the python used to build it. It is strongly reccomended to use python3 although python2.6+ is supported.
 You can also build the executable without any version info or metadata by using:
@@ -134,9 +141,9 @@ Then simply type this
 ## General Options:
    -h, --help                       Print this help text and exit
    --version                        Print program version and exit
-    -U, --update                     [BROKEN] Update this program to latest
+    -U, --update                     Update this program to latest version. Make
-                                     version. Make sure that you have sufficient
+                                     sure that you have sufficient permissions
-                                     permissions (run with sudo if needed)
+                                     (run with sudo if needed)
    -i, --ignore-errors              Continue on download errors, for example to
                                     skip unavailable videos in a playlist
                                     (default) (Alias: --no-abort-on-error)
@@ -461,10 +468,6 @@ Then simply type this
                                     files in the current directory to debug
                                     problems
    --print-traffic                  Display sent and read HTTP traffic
    -C, --call-home                  [Broken] Contact the youtube-dlc server for
                                     debugging
    --no-call-home                   Do not contact the youtube-dlc server for
                                     debugging (default)
 ## Workarounds:
    --encoding ENCODING              Force the specified encoding (experimental)
@@ -537,6 +540,11 @@ Then simply type this
                                     bestvideo+bestaudio), output to given
                                     container format. One of mkv, mp4, ogg,
                                     webm, flv. Ignored if no merge is required
    --allow-unplayable-formats       Allow unplayable formats to be listed and
                                     downloaded. All video postprocessing will
                                     also be turned off
    --no-allow-unplayable-formats    Do not allow unplayable formats to be
                                     listed or downloaded (default)
 ## Subtitle Options:
    --write-subs                     Write subtitle file
@@ -8,7 +8,7 @@ from datetime import datetime
 exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
 old_version = locals()['__version__']
-old_version_list = old_version.replace('-', '.').split(".", 4)
+old_version_list = old_version.split(".", 4)
 old_ver = '.'.join(old_version_list[:3])
 old_rev = old_version_list[3] if len(old_version_list) > 3 else ''
@@ -1,6 +1,5 @@
 # Supported sites
 - **1tv**: Первый канал
 - **1up.com**
 - **20min**
 - **220.ro**
 - **23video**
@@ -394,6 +393,8 @@
 - **HungamaSong**
 - **Hypem**
 - **ign.com**
 - **IGNArticle**
 - **IGNVideo**
 - **IHeartRadio**
 - **iheartradio:podcast**
 - **imdb**: Internet Movie Database trailers
@@ -701,7 +702,6 @@
 - **parliamentlive.tv**: UK parliament videos
 - **Patreon**
 - **pbs**: Public Broadcasting Service (PBS) and member stations: PBS: Public Broadcasting Service, APT - Alabama Public Television (WBIQ), GPB/Georgia Public Broadcasting (WGTV), Mississippi Public Broadcasting (WMPN), Nashville Public Television (WNPT), WFSU-TV (WFSU), WSRE (WSRE), WTCI (WTCI), WPBA/Channel 30 (WPBA), Alaska Public Media (KAKM), Arizona PBS (KAET), KNME-TV/Channel 5 (KNME), Vegas PBS (KLVX), AETN/ARKANSAS ETV NETWORK (KETS), KET (WKLE), WKNO/Channel 10 (WKNO), LPB/LOUISIANA PUBLIC BROADCASTING (WLPB), OETA (KETA), Ozarks Public Television (KOZK), WSIU Public Broadcasting (WSIU), KEET TV (KEET), KIXE/Channel 9 (KIXE), KPBS San Diego (KPBS), KQED (KQED), KVIE Public Television (KVIE), PBS SoCal/KOCE (KOCE), ValleyPBS (KVPT), CONNECTICUT PUBLIC TELEVISION (WEDH), KNPB Channel 5 (KNPB), SOPTV (KSYS), Rocky Mountain PBS (KRMA), KENW-TV3 (KENW), KUED Channel 7 (KUED), Wyoming PBS (KCWC), Colorado Public Television / KBDI 12 (KBDI), KBYU-TV (KBYU), Thirteen/WNET New York (WNET), WGBH/Channel 2 (WGBH), WGBY (WGBY), NJTV Public Media NJ (WNJT), WLIW21 (WLIW), mpt/Maryland Public Television (WMPB), WETA Television and Radio (WETA), WHYY (WHYY), PBS 39 (WLVT), WVPT - Your Source for PBS and More! (WVPT), Howard University Television (WHUT), WEDU PBS (WEDU), WGCU Public Media (WGCU), WPBT2 (WPBT), WUCF TV (WUCF), WUFT/Channel 5 (WUFT), WXEL/Channel 42 (WXEL), WLRN/Channel 17 (WLRN), WUSF Public Broadcasting (WUSF), ETV (WRLK), UNC-TV (WUNC), PBS Hawaii - Oceanic Cable Channel 10 (KHET), Idaho Public Television (KAID), KSPS (KSPS), OPB (KOPB), KWSU/Channel 10 & KTNW/Channel 31 (KWSU), WILL-TV (WILL), Network Knowledge - WSEC/Springfield (WSEC), WTTW11 (WTTW), Iowa Public Television/IPTV (KDIN), Nine Network (KETC), PBS39 Fort Wayne (WFWA), WFYI Indianapolis (WFYI), Milwaukee Public Television (WMVS), WNIN (WNIN), WNIT Public Television (WNIT), WPT (WPNE), WVUT/Channel 22 (WVUT), WEIU/Channel 51 (WEIU), WQPT-TV (WQPT), WYCC PBS Chicago (WYCC), WIPB-TV (WIPB), WTIU (WTIU), CET  (WCET), ThinkTVNetwork (WPTD), WBGU-TV (WBGU), WGVU TV (WGVU), NET1 (KUON), Pioneer Public Television (KWCM), SDPB Television (KUSD), TPT (KTCA), KSMQ (KSMQ), KPTS/Channel 8 (KPTS), KTWU/Channel 11 (KTWU), East Tennessee PBS (WSJK), WCTE-TV (WCTE), WLJT, Channel 11 (WLJT), WOSU TV (WOSU), WOUB/WOUC (WOUB), WVPB (WVPB), WKYU-PBS (WKYU), KERA 13 (KERA), MPBN (WCBB), Mountain Lake PBS (WCFE), NHPTV (WENH), Vermont PBS (WETK), witf (WITF), WQED Multimedia (WQED), WMHT Educational Telecommunications (WMHT), Q-TV (WDCQ), WTVS Detroit Public TV (WTVS), CMU Public Television (WCMU), WKAR-TV (WKAR), WNMU-TV Public TV 13 (WNMU), WDSE - WRPT (WDSE), WGTE TV (WGTE), Lakeland Public Television (KAWE), KMOS-TV - Channels 6.1, 6.2 and 6.3 (KMOS), MontanaPBS (KUSM), KRWG/Channel 22 (KRWG), KACV (KACV), KCOS/Channel 13 (KCOS), WCNY/Channel 24 (WCNY), WNED (WNED), WPBS (WPBS), WSKG Public TV (WSKG), WXXI (WXXI), WPSU (WPSU), WVIA Public Media Studios (WVIA), WTVI (WTVI), Western Reserve PBS (WNEO), WVIZ/PBS ideastream (WVIZ), KCTS 9 (KCTS), Basin PBS (KPBT), KUHT / Channel 8 (KUHT), KLRN (KLRN), KLRU (KLRU), WTJX Channel 12 (WTJX), WCVE PBS (WCVE), KBTC Public Television (KBTC)
 - **pcmag**
 - **PearVideo**
 - **PeerTube**
 - **People**
@@ -1218,9 +1218,9 @@
 - **youtube:history**: Youtube watch history, ":ythistory" for short (requires authentication)
 - **youtube:playlist**: YouTube.com playlists
 - **youtube:recommended**: YouTube.com recommended videos, ":ytrec" for short (requires authentication)
- - **youtube:search**: YouTube.com searches
+ - **youtube:search**: YouTube.com searches, "ytsearch" keyword
 - **youtube:search:date**: YouTube.com searches, newest videos first, "ytsearchdate" keyword
- - **youtube:search_url**: YouTube.com searches, "ytsearch" keyword
+ - **youtube:search_url**: YouTube.com search URLs
 - **youtube:subscriptions**: YouTube.com subscriptions feed, ":ytsubs" for short (requires authentication)
 - **youtube:tab**: YouTube.com tab
 - **youtube:watchlater**: Youtube watch later list, ":ytwatchlater" for short (requires authentication)
@@ -3,7 +3,7 @@
 from __future__ import unicode_literals
 import sys
-import os
+# import os
 import platform
 from PyInstaller.utils.win32.versioninfo import (
@@ -18,16 +18,15 @@ print('Building %sbit version' % arch)
 _x86 = '_x86' if arch == '32' else ''
 FILE_DESCRIPTION = 'Media Downloader%s' % (' (32 Bit)' if _x86 else '')
 SHORT_URLS = {'32': 'git.io/JUGsM', '64': 'git.io/JLh7K'}
-root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
+# root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
-print('Changing working directory to %s' % root_dir)
+# print('Changing working directory to %s' % root_dir)
-os.chdir(root_dir)
+# os.chdir(root_dir)
 exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
 VERSION = locals()['__version__']
-VERSION_LIST = VERSION.replace('-', '.').split('.')
+VERSION_LIST = VERSION.split('.')
 VERSION_LIST = list(map(int, VERSION_LIST)) + [0] * (4 - len(VERSION_LIST))
 print('Version: %s%s' % (VERSION, _x86))
@@ -49,7 +48,7 @@ VERSION_FILE = VSVersionInfo(
            StringTable(
                '040904B0', [
                    StringStruct('Comments', 'Youtube-dlc%s Command Line Interface.' % _x86),
-                    StringStruct('CompanyName', 'pukkandan@gmail.com'),
+                    StringStruct('CompanyName', 'https://github.com/pukkandan/yt-dlp'),
                    StringStruct('FileDescription', FILE_DESCRIPTION),
                    StringStruct('FileVersion', VERSION),
                    StringStruct('InternalName', 'youtube-dlc%s' % _x86),
@@ -59,7 +58,7 @@ VERSION_FILE = VSVersionInfo(
                    ),
                    StringStruct('OriginalFilename', 'youtube-dlc%s.exe' % _x86),
                    StringStruct('ProductName', 'Youtube-dlc%s' % _x86),
-                    StringStruct('ProductVersion', '%s%s | %s' % (VERSION, _x86, SHORT_URLS[arch])),
+                    StringStruct('ProductVersion', '%s%s' % (VERSION, _x86)),
                ])]),
        VarFileInfo([VarStruct('Translation', [0, 1200])])
    ]
@@ -73,6 +72,7 @@ PyInstaller.__main__.run([
    '--exclude-module=test',
    '--exclude-module=ytdlp_plugins',
    '--hidden-import=mutagen',
    '--hidden-import=Crypto',
    'youtube_dlc/__main__.py',
 ])
 SetVersion('dist/youtube-dlc%s.exe' % _x86, VERSION_FILE)
@@ -1 +1,2 @@
 mutagen
 pycryptodome
@@ -20,7 +20,7 @@ LONG_DESCRIPTION = '\n\n'.join((
    '**PS**: Many links in this document will not work since this is a copy of the README.md from Github',
    open("README.md", "r", encoding="utf-8").read()))
-REQUIREMENTS = ['mutagen']
+REQUIREMENTS = ['mutagen', 'pycryptodome']
 if len(sys.argv) >= 2 and sys.argv[1] == 'py2exe':
@@ -19,55 +19,46 @@ from youtube_dlc.compat import compat_str, compat_urlretrieve
 _TESTS = [
    (
        'https://s.ytimg.com/yts/jsbin/html5player-vflHOr_nV.js',
        'js',
        86,
        '>=<;:/.-[+*)(\'&%$#"!ZYX0VUTSRQPONMLKJIHGFEDCBA\\yxwvutsrqponmlkjihgfedcba987654321',
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-vfldJ8xgI.js',
        'js',
        85,
        '3456789a0cdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRS[UVWXYZ!"#$%&\'()*+,-./:;<=>?@',
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-vfle-mVwz.js',
        'js',
        90,
        ']\\[@?>=<;:/.-,+*)(\'&%$#"hZYXWVUTSRQPONMLKJIHGFEDCBAzyxwvutsrqponmlkjiagfedcb39876',
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-en_US-vfl0Cbn9e.js',
        'js',
        84,
        'O1I3456789abcde0ghijklmnopqrstuvwxyzABCDEFGHfJKLMN2PQRSTUVW@YZ!"#$%&\'()*+,-./:;<=',
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflXGBaUN.js',
        'js',
        '2ACFC7A61CA478CD21425E5A57EBD73DDC78E22A.2094302436B2D377D14A3BBA23022D023B8BC25AA',
        'A52CB8B320D22032ABB3A41D773D2B6342034902.A22E87CDD37DBE75A5E52412DC874AC16A7CFCA2',
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflBb0OQx.js',
        'js',
        84,
        '123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQ0STUVWXYZ!"#$%&\'()*+,@./:;<=>'
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-en_US-vfl9FYC6l.js',
        'js',
        83,
        '123456789abcdefghijklmnopqr0tuvwxyzABCDETGHIJKLMNOPQRS>UVWXYZ!"#$%&\'()*+,-./:;<=F'
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflCGk6yw/html5player.js',
        'js',
        '4646B5181C6C3020DF1D9C7FCFEA.AD80ABF70C39BD369CCCAE780AFBB98FA6B6CB42766249D9488C288',
        '82C8849D94266724DC6B6AF89BBFA087EACCD963.B93C07FBA084ACAEFCF7C9D1FD0203C6C1815B6B'
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflKjOTVq/html5player.js',
        'js',
        '312AA52209E3623129A412D56A40F11CB0AF14AE.3EE09501CB14E3BCDC3B2AE808BF3F1D14E7FBF12',
        '112AA5220913623229A412D56A40F11CB0AF14AE.3EE0950FCB14EEBCDC3B2AE808BF331D14E7FBF3',
    )
@@ -78,6 +69,10 @@ class TestPlayerInfo(unittest.TestCase):
    def test_youtube_extract_player_info(self):
        PLAYER_URLS = (
            ('https://www.youtube.com/s/player/64dddad9/player_ias.vflset/en_US/base.js', '64dddad9'),
            ('https://www.youtube.com/s/player/64dddad9/player_ias.vflset/fr_FR/base.js', '64dddad9'),
            ('https://www.youtube.com/s/player/64dddad9/player-plasma-ias-phone-en_US.vflset/base.js', '64dddad9'),
            ('https://www.youtube.com/s/player/64dddad9/player-plasma-ias-phone-de_DE.vflset/base.js', '64dddad9'),
            ('https://www.youtube.com/s/player/64dddad9/player-plasma-ias-tablet-en_US.vflset/base.js', '64dddad9'),
            # obsolete
            ('https://www.youtube.com/yts/jsbin/player_ias-vfle4-e03/en_US/base.js', 'vfle4-e03'),
            ('https://www.youtube.com/yts/jsbin/player_ias-vfl49f_g4/en_US/base.js', 'vfl49f_g4'),
@@ -100,13 +95,13 @@ class TestSignature(unittest.TestCase):
            os.mkdir(self.TESTDATA_DIR)
-def make_tfunc(url, stype, sig_input, expected_sig):
+def make_tfunc(url, sig_input, expected_sig):
    m = re.match(r'.*-([a-zA-Z0-9_-]+)(?:/watch_as3|/html5player)?\.[a-z]+$', url)
    assert m, '%r should follow URL format' % url
    test_id = m.group(1)
    def test_func(self):
-        basename = 'player-%s.%s' % (test_id, stype)
+        basename = 'player-%s.js' % test_id
        fn = os.path.join(self.TESTDATA_DIR, basename)
        if not os.path.exists(fn):
@@ -114,22 +109,16 @@ def make_tfunc(url, stype, sig_input, expected_sig):
        ydl = FakeYDL()
        ie = YoutubeIE(ydl)
-        if stype == 'js':
+        with io.open(fn, encoding='utf-8') as testf:
-            with io.open(fn, encoding='utf-8') as testf:
+            jscode = testf.read()
-                jscode = testf.read()
+        func = ie._parse_sig_js(jscode)
            func = ie._parse_sig_js(jscode)
        else:
            assert stype == 'swf'
            with open(fn, 'rb') as testf:
                swfcode = testf.read()
            func = ie._parse_sig_swf(swfcode)
        src_sig = (
            compat_str(string.printable[:sig_input])
            if isinstance(sig_input, int) else sig_input)
        got_sig = func(src_sig)
        self.assertEqual(got_sig, expected_sig)
-    test_func.__name__ = str('test_signature_' + stype + '_' + test_id)
+    test_func.__name__ = str('test_signature_js_' + test_id)
    setattr(TestSignature, test_func.__name__, test_func)
@@ -27,6 +27,7 @@ import traceback
 import random
 from string import ascii_letters
 from zipimport import zipimporter
 from .compat import (
    compat_basestring,
@@ -179,6 +180,7 @@ class YoutubeDL(object):
                       of 'skip_download' or 'simulate'.
    simulate:          Do not download the video files.
    format:            Video format code. see "FORMAT SELECTION" for more details.
    allow_unplayable_formats:   Allow unplayable formats to be extracted and downloaded.
    format_sort:       How to sort the video formats. see "Sorting Formats"
                       for more details.
    format_sort_force: Force the given format_sort. see "Sorting Formats"
@@ -2131,7 +2133,7 @@ class YoutubeDL(object):
                sub_format = sub_info['ext']
                sub_fn = self.prepare_filename(info_dict, 'subtitle')
                sub_filename = subtitles_filename(
-                     temp_filename if not skip_dl else sub_fn,
+                    temp_filename if not skip_dl else sub_fn,
                    sub_lang, sub_format, info_dict.get('ext'))
                sub_filename_final = subtitles_filename(sub_fn, sub_lang, sub_format, info_dict.get('ext'))
                if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(sub_filename)):
@@ -2291,13 +2293,15 @@ class YoutubeDL(object):
                if info_dict.get('requested_formats') is not None:
                    downloaded = []
                    merger = FFmpegMergerPP(self)
-                    if not merger.available:
+                    if self.params.get('allow_unplayable_formats'):
-                        postprocessors = []
+                        self.report_warning(
-                        self.report_warning('You have requested multiple '
+                            'You have requested merging of multiple formats '
-                                            'formats but ffmpeg is not installed.'
+                            'while also allowing unplayable formats to be downloaded. '
-                                            ' The formats won\'t be merged.')
+                            'The formats won\'t be merged to prevent data corruption.')
-                    else:
+                    elif not merger.available:
-                        postprocessors = [merger]
+                        self.report_warning(
                            'You have requested merging of multiple formats but ffmpeg is not installed. '
                            'The formats won\'t be merged.')
                    def compatible_formats(formats):
                        # TODO: some formats actually allow this (mkv, webm, ogg, mp4), but not all of them.
@@ -2337,6 +2341,7 @@ class YoutubeDL(object):
                    full_filename = correct_ext(full_filename)
                    temp_filename = correct_ext(temp_filename)
                    dl_filename = existing_file(full_filename, temp_filename)
                    info_dict['__real_download'] = False
                    if dl_filename is None:
                        for f in requested_formats:
                            new_info = dict(info_dict)
@@ -2348,11 +2353,13 @@ class YoutubeDL(object):
                                return
                            downloaded.append(fname)
                            partial_success, real_download = dl(fname, new_info)
                            info_dict['__real_download'] = info_dict['__real_download'] or real_download
                            success = success and partial_success
-                        info_dict['__postprocessors'] = postprocessors
+                        if merger.available and not self.params.get('allow_unplayable_formats'):
-                        info_dict['__files_to_merge'] = downloaded
+                            info_dict['__postprocessors'].append(merger)
-                        # Even if there were no downloads, it is being merged only now
+                            info_dict['__files_to_merge'] = downloaded
-                        info_dict['__real_download'] = True
+                            # Even if there were no downloads, it is being merged only now
                            info_dict['__real_download'] = True
                else:
                    # Just a single file
                    dl_filename = existing_file(full_filename, temp_filename)
@@ -2683,7 +2690,7 @@ class YoutubeDL(object):
                    '|',
                    format_field(f, 'filesize', ' %s', func=format_bytes) + format_field(f, 'filesize_approx', '~%s', func=format_bytes),
                    format_field(f, 'tbr', '%4dk'),
-                    f.get('protocol').replace('http_dash_segments', 'dash').replace("native", "n"),
+                    f.get('protocol').replace('http_dash_segments', 'dash').replace("native", "n").replace('niconico_', ''),
                    '|',
                    format_field(f, 'vcodec', default='unknown').replace('none', ''),
                    format_field(f, 'vbr', '%4dk'),
@@ -2764,7 +2771,12 @@ class YoutubeDL(object):
                self.get_encoding()))
        write_string(encoding_str, encoding=None)
-        self._write_string('[debug] yt-dlp version %s\n' % __version__)
+        source = (
            '(exe)' if hasattr(sys, 'frozen')
            else '(zip)' if isinstance(globals().get('__loader__'), zipimporter)
            else '(source)' if os.path.basename(sys.argv[0]) == '__main__.py'
            else '')
        self._write_string('[debug] yt-dlp version %s %s\n' % (__version__, source))
        if _LAZY_LOADER:
            self._write_string('[debug] Lazy loading extractors enabled\n')
        if _PLUGIN_CLASSES:
@@ -2791,8 +2803,10 @@ class YoutubeDL(object):
                return impl_name + ' version %d.%d.%d' % sys.pypy_version_info[:3]
            return impl_name
-        self._write_string('[debug] Python version %s (%s) - %s\n' % (
+        self._write_string('[debug] Python version %s (%s %s) - %s\n' % (
-            platform.python_version(), python_implementation(),
+            platform.python_version(),
            python_implementation(),
            platform.architecture()[0],
            platform_name()))
        exe_versions = FFmpegPostProcessor.get_versions(self)
@@ -2895,20 +2909,17 @@ class YoutubeDL(object):
        return encoding
    def _write_thumbnails(self, info_dict, filename):  # return the extensions
-        if self.params.get('writethumbnail', False):
+        write_all = self.params.get('write_all_thumbnails', False)
-            thumbnails = info_dict.get('thumbnails')
+        thumbnails = []
-            if thumbnails:
+        if write_all or self.params.get('writethumbnail', False):
                thumbnails = [thumbnails[-1]]
        elif self.params.get('write_all_thumbnails', False):
            thumbnails = info_dict.get('thumbnails') or []
-        else:
+        multiple = write_all and len(thumbnails) > 1
            thumbnails = []
        ret = []
-        for t in thumbnails:
+        for t in thumbnails[::1 if write_all else -1]:
            thumb_ext = determine_ext(t['url'], 'jpg')
-            suffix = '%s.' % t['id'] if len(thumbnails) > 1 else ''
+            suffix = '%s.' % t['id'] if multiple else ''
-            thumb_display_id = '%s ' % t['id'] if len(thumbnails) > 1 else ''
+            thumb_display_id = '%s ' % t['id'] if multiple else ''
            t['filename'] = thumb_filename = replace_extension(filename, suffix + thumb_ext, info_dict.get('ext'))
            if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(thumb_filename)):
@@ -2928,4 +2939,6 @@ class YoutubeDL(object):
                except (compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
                    self.report_warning('Unable to download thumbnail "%s": %s' %
                                        (t['url'], error_to_compat_str(err)))
            if ret and not write_all:
                break
        return ret
@@ -15,7 +15,6 @@ import sys
 from .options import (
    parseOpts,
    _remux_formats,
 )
 from .compat import (
    compat_getpass,
@@ -24,7 +23,6 @@ from .compat import (
 from .utils import (
    DateRange,
    decodeOption,
    DEFAULT_OUTTMPL,
    DownloadError,
    ExistingVideoReached,
    expand_path,
@@ -33,11 +31,12 @@ from .utils import (
    preferredencoding,
    read_batch_urls,
    RejectedVideoReached,
    REMUX_EXTENSIONS,
    render_table,
    SameFileError,
    setproctitle,
    std_headers,
    write_string,
    render_table,
 )
 from .update import update_self
 from .downloader import (
@@ -211,13 +210,12 @@ def _real_main(argv=None):
        if not opts.audioquality.isdigit():
            parser.error('invalid audio quality specified')
    if opts.recodevideo is not None:
-        if opts.recodevideo not in _remux_formats:
+        if opts.recodevideo not in REMUX_EXTENSIONS:
            parser.error('invalid video recode format specified')
    if opts.remuxvideo and opts.recodevideo:
        opts.remuxvideo = None
        write_string('WARNING: --remux-video is ignored since --recode-video was given\n', out=sys.stderr)
    if opts.remuxvideo is not None:
-        if opts.remuxvideo not in _remux_formats:
+        opts.remuxvideo = opts.remuxvideo.replace(' ', '')
        remux_regex = r'{0}(?:/{0})*$'.format(r'(?:\w+>)?(?:%s)' % '|'.join(REMUX_EXTENSIONS))
        if not re.match(remux_regex, opts.remuxvideo):
            parser.error('invalid video remux format specified')
    if opts.convertsubtitles is not None:
        if opts.convertsubtitles not in ['srt', 'vtt', 'ass', 'lrc']:
@@ -232,11 +230,6 @@ def _real_main(argv=None):
    if opts.extractaudio and not opts.keepvideo and opts.format is None:
        opts.format = 'bestaudio/best'
    # --all-sub automatically sets --write-sub if --write-auto-sub is not given
    # this was the old behaviour if only --all-sub was given.
    if opts.allsubtitles and not opts.writeautomaticsub:
        opts.writesubtitles = True
    outtmpl = opts.outtmpl
    if not outtmpl:
        outtmpl = {'default': (
@@ -269,6 +262,40 @@ def _real_main(argv=None):
    any_printing = opts.print_json
    download_archive_fn = expand_path(opts.download_archive) if opts.download_archive is not None else opts.download_archive
    def report_conflict(arg1, arg2):
        write_string('WARNING: %s is ignored since %s was given\n' % (arg2, arg1), out=sys.stderr)
    if opts.remuxvideo and opts.recodevideo:
        report_conflict('--recode-video', '--remux-video')
        opts.remuxvideo = False
    if opts.allow_unplayable_formats:
        if opts.extractaudio:
            report_conflict('--allow-unplayable-formats', '--extract-audio')
            opts.extractaudio = False
        if opts.remuxvideo:
            report_conflict('--allow-unplayable-formats', '--remux-video')
            opts.remuxvideo = False
        if opts.recodevideo:
            report_conflict('--allow-unplayable-formats', '--recode-video')
            opts.recodevideo = False
        if opts.addmetadata:
            report_conflict('--allow-unplayable-formats', '--add-metadata')
            opts.addmetadata = False
        if opts.embedsubtitles:
            report_conflict('--allow-unplayable-formats', '--embed-subs')
            opts.embedsubtitles = False
        if opts.embedthumbnail:
            report_conflict('--allow-unplayable-formats', '--embed-thumbnail')
            opts.embedthumbnail = False
        if opts.xattrs:
            report_conflict('--allow-unplayable-formats', '--xattrs')
            opts.xattrs = False
        if opts.fixup and opts.fixup.lower() not in ('never', 'ignore'):
            report_conflict('--allow-unplayable-formats', '--fixup')
        opts.fixup = 'never'
        if opts.sponskrub:
            report_conflict('--allow-unplayable-formats', '--sponskrub')
        opts.sponskrub = False
    # PostProcessors
    postprocessors = []
    if opts.metafromfield:
@@ -310,9 +337,17 @@ def _real_main(argv=None):
            'format': opts.convertsubtitles,
        })
    if opts.embedsubtitles:
        already_have_subtitle = opts.writesubtitles
        postprocessors.append({
            'key': 'FFmpegEmbedSubtitle',
            'already_have_subtitle': already_have_subtitle
        })
        if not already_have_subtitle:
            opts.writesubtitles = True
    # --all-sub automatically sets --write-sub if --write-auto-sub is not given
    # this was the old behaviour if only --all-sub was given.
    if opts.allsubtitles and not opts.writeautomaticsub:
        opts.writesubtitles = True
    if opts.embedthumbnail:
        already_have_thumbnail = opts.writethumbnail or opts.write_all_thumbnails
        postprocessors.append({
@@ -344,16 +379,23 @@ def _real_main(argv=None):
            'when': 'aftermove'
        })
-    _args_compat_warning = 'WARNING: %s given without specifying name. The arguments will be given to all %s\n'
+    def report_args_compat(arg, name):
        write_string(
            'WARNING: %s given without specifying name. The arguments will be given to all %s\n' % (arg, name),
            out=sys.stderr)
    if 'default' in opts.external_downloader_args:
-        write_string(_args_compat_warning % ('--external-downloader-args', 'external downloaders'), out=sys.stderr),
+        report_args_compat('--external-downloader-args', 'external downloaders')
    if 'default-compat' in opts.postprocessor_args and 'default' not in opts.postprocessor_args:
-        write_string(_args_compat_warning % ('--post-processor-args', 'post-processors'), out=sys.stderr),
+        report_args_compat('--post-processor-args', 'post-processors')
        opts.postprocessor_args.setdefault('sponskrub', [])
        opts.postprocessor_args['default'] = opts.postprocessor_args['default-compat']
-    audio_ext = opts.audioformat if (opts.extractaudio and opts.audioformat != 'best') else None
+    final_ext = (
        opts.recodevideo
        or (opts.remuxvideo in REMUX_EXTENSIONS) and opts.remuxvideo
        or (opts.extractaudio and opts.audioformat != 'best') and opts.audioformat
        or None)
    match_filter = (
        None if opts.match_filter is None
@@ -385,6 +427,7 @@ def _real_main(argv=None):
        'simulate': opts.simulate or any_getting,
        'skip_download': opts.skip_download,
        'format': opts.format,
        'allow_unplayable_formats': opts.allow_unplayable_formats,
        'format_sort': opts.format_sort,
        'format_sort_force': opts.format_sort_force,
        'allow_multiple_video_streams': opts.allow_multiple_video_streams,
@@ -423,6 +466,7 @@ def _real_main(argv=None):
        'writedescription': opts.writedescription,
        'writeannotations': opts.writeannotations,
        'writeinfojson': opts.writeinfojson or opts.getcomments,
        'allow_playlist_files': opts.allow_playlist_files,
        'getcomments': opts.getcomments,
        'writethumbnail': opts.writethumbnail,
        'write_all_thumbnails': opts.write_all_thumbnails,
@@ -474,7 +518,7 @@ def _real_main(argv=None):
        'extract_flat': opts.extract_flat,
        'mark_watched': opts.mark_watched,
        'merge_output_format': opts.merge_output_format,
-        'final_ext': opts.recodevideo or opts.remuxvideo or audio_ext,
+        'final_ext': final_ext,
        'postprocessors': postprocessors,
        'fixup': opts.fixup,
        'source_address': opts.source_address,
@@ -505,16 +549,22 @@ def _real_main(argv=None):
    }
    with YoutubeDL(ydl_opts) as ydl:
-        # Update version
+        actual_use = len(all_urls) or opts.load_info_filename
        if opts.update_self:
            update_self(ydl.to_screen, opts.verbose, ydl._opener)
        # Remove cache dir
        if opts.rm_cachedir:
            ydl.cache.remove()
        # Update version
        if opts.update_self:
            # If updater returns True, exit. Required for windows
            if update_self(ydl.to_screen, opts.verbose, ydl._opener):
                if actual_use:
                    parser.error('The program must exit for the update to complete')
                sys.exit()
        # Maybe do nothing
-        if (len(all_urls) < 1) and (opts.load_info_filename is None):
+        if not actual_use:
            if opts.update_self or opts.rm_cachedir:
                sys.exit()
@@ -1,23 +1,33 @@
 from __future__ import unicode_literals
 from ..utils import (
    determine_protocol,
 )
 def _get_real_downloader(info_dict, protocol=None, *args, **kwargs):
    info_copy = info_dict.copy()
    if protocol:
        info_copy['protocol'] = protocol
    return get_suitable_downloader(info_copy, *args, **kwargs)
 # Some of these require _get_real_downloader
 from .common import FileDownloader
 from .dash import DashSegmentsFD
 from .f4m import F4mFD
 from .hls import HlsFD
 from .http import HttpFD
 from .rtmp import RtmpFD
 from .dash import DashSegmentsFD
 from .rtsp import RtspFD
 from .ism import IsmFD
 from .niconico import NiconicoDmcFD
 from .youtube_live_chat import YoutubeLiveChatReplayFD
 from .external import (
    get_external_downloader,
    FFmpegFD,
 )
 from ..utils import (
    determine_protocol,
 )
 PROTOCOL_MAP = {
    'rtmp': RtmpFD,
    'm3u8_native': HlsFD,
@@ -27,11 +37,12 @@ PROTOCOL_MAP = {
    'f4m': F4mFD,
    'http_dash_segments': DashSegmentsFD,
    'ism': IsmFD,
    'niconico_dmc': NiconicoDmcFD,
    'youtube_live_chat_replay': YoutubeLiveChatReplayFD,
 }
-def get_suitable_downloader(info_dict, params={}):
+def get_suitable_downloader(info_dict, params={}, default=HttpFD):
    """Get the downloader class that can handle the info dict."""
    protocol = determine_protocol(info_dict)
    info_dict['protocol'] = protocol
@@ -45,16 +56,17 @@ def get_suitable_downloader(info_dict, params={}):
        if ed.can_download(info_dict):
            return ed
-    if protocol.startswith('m3u8') and info_dict.get('is_live'):
+    if protocol.startswith('m3u8'):
-        return FFmpegFD
+        if info_dict.get('is_live'):
            return FFmpegFD
        elif _get_real_downloader(info_dict, 'frag_urls', params, None):
            return HlsFD
        elif params.get('hls_prefer_native') is True:
            return HlsFD
        elif params.get('hls_prefer_native') is False:
            return FFmpegFD
-    if protocol == 'm3u8' and params.get('hls_prefer_native') is True:
+    return PROTOCOL_MAP.get(protocol, default)
        return HlsFD
    if protocol == 'm3u8_native' and params.get('hls_prefer_native') is False:
        return FFmpegFD
    return PROTOCOL_MAP.get(protocol, HttpFD)
 __all__ = [
@@ -1,6 +1,8 @@
 from __future__ import unicode_literals
 from ..downloader import _get_real_downloader
 from .fragment import FragmentFD
 from ..compat import compat_urllib_error
 from ..utils import (
    DownloadError,
@@ -20,31 +22,42 @@ class DashSegmentsFD(FragmentFD):
        fragments = info_dict['fragments'][:1] if self.params.get(
            'test', False) else info_dict['fragments']
        real_downloader = _get_real_downloader(info_dict, 'frag_urls', self.params, None)
        ctx = {
            'filename': filename,
            'total_frags': len(fragments),
        }
-        self._prepare_and_start_frag_download(ctx)
+        if real_downloader:
            self._prepare_external_frag_download(ctx)
        else:
            self._prepare_and_start_frag_download(ctx)
        fragment_retries = self.params.get('fragment_retries', 0)
        skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
        fragment_urls = []
        frag_index = 0
        for i, fragment in enumerate(fragments):
            frag_index += 1
            if frag_index <= ctx['fragment_index']:
                continue
            fragment_url = fragment.get('url')
            if not fragment_url:
                assert fragment_base_url
                fragment_url = urljoin(fragment_base_url, fragment['path'])
            if real_downloader:
                fragment_urls.append(fragment_url)
                continue
            # In DASH, the first segment contains necessary headers to
            # generate a valid MP4 file, so always abort for the first segment
            fatal = i == 0 or not skip_unavailable_fragments
            count = 0
            while count <= fragment_retries:
                try:
                    fragment_url = fragment.get('url')
                    if not fragment_url:
                        assert fragment_base_url
                        fragment_url = urljoin(fragment_base_url, fragment['path'])
                    success, frag_content = self._download_fragment(ctx, fragment_url, info_dict)
                    if not success:
                        return False
@@ -75,6 +88,16 @@ class DashSegmentsFD(FragmentFD):
                self.report_error('giving up after %s fragment retries' % fragment_retries)
                return False
-        self._finish_frag_download(ctx)
+        if real_downloader:
-
+            info_copy = info_dict.copy()
            info_copy['url_list'] = fragment_urls
            fd = real_downloader(self.ydl, self.params)
            # TODO: Make progress updates work without hooking twice
            # for ph in self._progress_hooks:
            #     fd.add_progress_hook(ph)
            success = fd.real_download(filename, info_copy)
            if not success:
                return False
        else:
            self._finish_frag_download(ctx)
        return True
@@ -6,6 +6,12 @@ import subprocess
 import sys
 import time
 try:
    from Crypto.Cipher import AES
    can_decrypt_frag = True
 except ImportError:
    can_decrypt_frag = False
 from .common import FileDownloader
 from ..compat import (
    compat_setenv,
@@ -18,15 +24,20 @@ from ..utils import (
    cli_bool_option,
    cli_configuration_args,
    encodeFilename,
    error_to_compat_str,
    encodeArgument,
    handle_youtubedl_headers,
    check_executable,
    is_outdated_version,
    process_communicate_or_kill,
    sanitized_Request,
    sanitize_open,
 )
 class ExternalFD(FileDownloader):
    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps')
    def real_download(self, filename, info_dict):
        self.report_destination(filename)
        tmpfilename = self.temp_name(filename)
@@ -79,7 +90,7 @@ class ExternalFD(FileDownloader):
    @classmethod
    def supports(cls, info_dict):
-        return info_dict['protocol'] in ('http', 'https', 'ftp', 'ftps')
+        return info_dict['protocol'] in cls.SUPPORTED_PROTOCOLS
    @classmethod
    def can_download(cls, info_dict):
@@ -109,8 +120,53 @@ class ExternalFD(FileDownloader):
        _, stderr = process_communicate_or_kill(p)
        if p.returncode != 0:
            self.to_stderr(stderr.decode('utf-8', 'replace'))
        if 'url_list' in info_dict:
            file_list = []
            for [i, url] in enumerate(info_dict['url_list']):
                tmpsegmentname = '%s_%s.frag' % (tmpfilename, i)
                file_list.append(tmpsegmentname)
            key_list = info_dict.get('key_list')
            decrypt_info = None
            dest, _ = sanitize_open(tmpfilename, 'wb')
            for i, file in enumerate(file_list):
                src, _ = sanitize_open(file, 'rb')
                if key_list:
                    decrypt_info = next((x for x in key_list if x['INDEX'] == i), decrypt_info)
                    if decrypt_info['METHOD'] == 'AES-128':
                        iv = decrypt_info.get('IV')
                        decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
                            self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
                        encrypted_data = src.read()
                        decrypted_data = AES.new(
                            decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(encrypted_data)
                        dest.write(decrypted_data)
                    else:
                        fragment_data = src.read()
                        dest.write(fragment_data)
                else:
                    fragment_data = src.read()
                    dest.write(fragment_data)
                src.close()
            dest.close()
            if not self.params.get('keep_fragments', False):
                for file_path in file_list:
                    try:
                        os.remove(file_path)
                    except OSError as ose:
                        self.report_error("Unable to delete file %s; %s" % (file_path, error_to_compat_str(ose)))
                try:
                    file_path = '%s.frag.urls' % tmpfilename
                    os.remove(file_path)
                except OSError as ose:
                    self.report_error("Unable to delete file %s; %s" % (file_path, error_to_compat_str(ose)))
        return p.returncode
    def _prepare_url(self, info_dict, url):
        headers = info_dict.get('http_headers')
        return sanitized_Request(url, None, headers) if headers else url
 class CurlFD(ExternalFD):
    AVAILABLE_OPT = '-V'
@@ -186,15 +242,17 @@ class WgetFD(ExternalFD):
 class Aria2cFD(ExternalFD):
    AVAILABLE_OPT = '-v'
    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'frag_urls')
    def _make_cmd(self, tmpfilename, info_dict):
        cmd = [self.exe, '-c']
        cmd += self._configuration_args([
            '--min-split-size', '1M', '--max-connection-per-server', '4'])
        dn = os.path.dirname(tmpfilename)
        if 'url_list' not in info_dict:
            cmd += ['--out', os.path.basename(tmpfilename)]
        verbose_level_args = ['--console-log-level=warn', '--summary-interval=0']
        cmd += self._configuration_args(['--file-allocation=none', '-x16', '-j16', '-s16'] + verbose_level_args)
        if dn:
            cmd += ['--dir', dn]
        cmd += ['--out', os.path.basename(tmpfilename)]
        if info_dict.get('http_headers') is not None:
            for key, val in info_dict['http_headers'].items():
                cmd += ['--header', '%s: %s' % (key, val)]
@@ -202,7 +260,22 @@ class Aria2cFD(ExternalFD):
        cmd += self._option('--all-proxy', 'proxy')
        cmd += self._bool_option('--check-certificate', 'nocheckcertificate', 'false', 'true', '=')
        cmd += self._bool_option('--remote-time', 'updatetime', 'true', 'false', '=')
-        cmd += ['--', info_dict['url']]
+        cmd += ['--auto-file-renaming=false']
        if 'url_list' in info_dict:
            cmd += verbose_level_args
            cmd += ['--uri-selector', 'inorder', '--download-result=hide']
            url_list_file = '%s.frag.urls' % tmpfilename
            url_list = []
            for [i, url] in enumerate(info_dict['url_list']):
                tmpsegmentname = '%s_%s.frag' % (os.path.basename(tmpfilename), i)
                url_list.append('%s\n\tout=%s' % (url, tmpsegmentname))
            stream, _ = sanitize_open(url_list_file, 'wb')
            stream.write('\n'.join(url_list).encode('utf-8'))
            stream.close()
            cmd += ['-i', url_list_file]
        else:
            cmd += ['--', info_dict['url']]
        return cmd
@@ -221,9 +294,7 @@ class HttpieFD(ExternalFD):
 class FFmpegFD(ExternalFD):
-    @classmethod
+    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'm3u8', 'rtsp', 'rtmp', 'mms')
    def supports(cls, info_dict):
        return info_dict['protocol'] in ('http', 'https', 'ftp', 'ftps', 'm3u8', 'rtsp', 'rtmp', 'mms')
    @classmethod
    def available(cls):
@@ -267,13 +267,14 @@ class F4mFD(FragmentFD):
        media = doc.findall(_add_ns('media'))
        if not media:
            self.report_error('No media found')
-        for e in (doc.findall(_add_ns('drmAdditionalHeader'))
+        if not self.params.get('allow_unplayable_formats'):
-                  + doc.findall(_add_ns('drmAdditionalHeaderSet'))):
+            for e in (doc.findall(_add_ns('drmAdditionalHeader'))
-            # If id attribute is missing it's valid for all media nodes
+                      + doc.findall(_add_ns('drmAdditionalHeaderSet'))):
-            # without drmAdditionalHeaderId or drmAdditionalHeaderSetId attribute
+                # If id attribute is missing it's valid for all media nodes
-            if 'id' not in e.attrib:
+                # without drmAdditionalHeaderId or drmAdditionalHeaderSetId attribute
-                self.report_error('Missing ID in f4m DRM')
+                if 'id' not in e.attrib:
-        media = remove_encrypted_media(media)
+                    self.report_error('Missing ID in f4m DRM')
            media = remove_encrypted_media(media)
        if not media:
            self.report_error('Unsupported DRM')
        return media
@@ -95,11 +95,12 @@ class FragmentFD(FileDownloader):
        frag_index_stream.write(json.dumps({'downloader': downloader}))
        frag_index_stream.close()
-    def _download_fragment(self, ctx, frag_url, info_dict, headers=None):
+    def _download_fragment(self, ctx, frag_url, info_dict, headers=None, request_data=None):
        fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], ctx['fragment_index'])
        fragment_info_dict = {
            'url': frag_url,
            'http_headers': headers or info_dict.get('http_headers'),
            'request_data': request_data,
        }
        success = ctx['dl'].download(fragment_filename, fragment_info_dict)
        if not success:
@@ -277,3 +278,24 @@ class FragmentFD(FileDownloader):
            'status': 'finished',
            'elapsed': elapsed,
        })
    def _prepare_external_frag_download(self, ctx):
        if 'live' not in ctx:
            ctx['live'] = False
        if not ctx['live']:
            total_frags_str = '%d' % ctx['total_frags']
            ad_frags = ctx.get('ad_frags', 0)
            if ad_frags:
                total_frags_str += ' (not including %d ad)' % ad_frags
        else:
            total_frags_str = 'unknown (live)'
        self.to_screen(
            '[%s] Total fragments: %s' % (self.FD_NAME, total_frags_str))
        tmpfilename = self.temp_name(ctx['filename'])
        # Should be initialized before ytdl file check
        ctx.update({
            'tmpfilename': tmpfilename,
            'fragment_index': 0,
        })
@@ -8,6 +8,7 @@ try:
 except ImportError:
    can_decrypt_frag = False
 from ..downloader import _get_real_downloader
 from .fragment import FragmentFD
 from .external import FFmpegFD
@@ -28,9 +29,8 @@ class HlsFD(FragmentFD):
    FD_NAME = 'hlsnative'
    @staticmethod
-    def can_download(manifest, info_dict):
+    def can_download(manifest, info_dict, allow_unplayable_formats=False):
-        UNSUPPORTED_FEATURES = (
+        UNSUPPORTED_FEATURES = [
            r'#EXT-X-KEY:METHOD=(?!NONE|AES-128)',  # encrypted streams [1]
            # r'#EXT-X-BYTERANGE',  # playlists composed of byte ranges of media files [2]
            # Live streams heuristic does not always work (e.g. geo restricted to Germany
@@ -49,7 +49,11 @@ class HlsFD(FragmentFD):
            # 3. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.2
            # 4. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.5
            # 5. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.5
-        )
+        ]
        if not allow_unplayable_formats:
            UNSUPPORTED_FEATURES += [
                r'#EXT-X-KEY:METHOD=(?!NONE|AES-128)',  # encrypted streams [1]
            ]
        check_results = [not re.search(feature, manifest) for feature in UNSUPPORTED_FEATURES]
        is_aes128_enc = '#EXT-X-KEY:METHOD=AES-128' in manifest
        check_results.append(can_decrypt_frag or not is_aes128_enc)
@@ -65,7 +69,7 @@ class HlsFD(FragmentFD):
        man_url = urlh.geturl()
        s = urlh.read().decode('utf-8', 'ignore')
-        if not self.can_download(s, info_dict):
+        if not self.can_download(s, info_dict, self.params.get('allow_unplayable_formats')):
            if info_dict.get('extra_param_to_segment_url') or info_dict.get('_decryption_key_url'):
                self.report_error('pycrypto not found. Please install it.')
                return False
@@ -73,10 +77,13 @@ class HlsFD(FragmentFD):
                'hlsnative has detected features it does not support, '
                'extraction will be delegated to ffmpeg')
            fd = FFmpegFD(self.ydl, self.params)
-            for ph in self._progress_hooks:
+            # TODO: Make progress updates work without hooking twice
-                fd.add_progress_hook(ph)
+            # for ph in self._progress_hooks:
            #     fd.add_progress_hook(ph)
            return fd.real_download(filename, info_dict)
        real_downloader = _get_real_downloader(info_dict, 'frag_urls', self.params, None)
        def is_ad_fragment_start(s):
            return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=ad' in s
                    or s.startswith('#UPLYNK-SEGMENT') and s.endswith(',ad'))
@@ -85,6 +92,8 @@ class HlsFD(FragmentFD):
            return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=master' in s
                    or s.startswith('#UPLYNK-SEGMENT') and s.endswith(',segment'))
        fragment_urls = []
        media_frags = 0
        ad_frags = 0
        ad_frag_next = False
@@ -109,7 +118,10 @@ class HlsFD(FragmentFD):
            'ad_frags': ad_frags,
        }
-        self._prepare_and_start_frag_download(ctx)
+        if real_downloader:
            self._prepare_external_frag_download(ctx)
        else:
            self._prepare_and_start_frag_download(ctx)
        fragment_retries = self.params.get('fragment_retries', 0)
        skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
@@ -122,6 +134,7 @@ class HlsFD(FragmentFD):
        i = 0
        media_sequence = 0
        decrypt_info = {'METHOD': 'NONE'}
        key_list = []
        byte_range = {}
        frag_index = 0
        ad_frag_next = False
@@ -140,6 +153,11 @@ class HlsFD(FragmentFD):
                        else compat_urlparse.urljoin(man_url, line))
                    if extra_query:
                        frag_url = update_url_query(frag_url, extra_query)
                    if real_downloader:
                        fragment_urls.append(frag_url)
                        continue
                    count = 0
                    headers = info_dict.get('http_headers', {})
                    if byte_range:
@@ -168,6 +186,7 @@ class HlsFD(FragmentFD):
                        self.report_error(
                            'giving up after %s fragment retries' % fragment_retries)
                        return False
                    if decrypt_info['METHOD'] == 'AES-128':
                        iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', media_sequence)
                        decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
@@ -197,6 +216,10 @@ class HlsFD(FragmentFD):
                            decrypt_info['URI'] = update_url_query(decrypt_info['URI'], extra_query)
                        if decrypt_url != decrypt_info['URI']:
                            decrypt_info['KEY'] = None
                    key_data = decrypt_info.copy()
                    key_data['INDEX'] = frag_index
                    key_list.append(key_data)
                elif line.startswith('#EXT-X-MEDIA-SEQUENCE'):
                    media_sequence = int(line[22:])
                elif line.startswith('#EXT-X-BYTERANGE'):
@@ -211,6 +234,17 @@ class HlsFD(FragmentFD):
                elif is_ad_fragment_end(line):
                    ad_frag_next = False
-        self._finish_frag_download(ctx)
+        if real_downloader:
-
+            info_copy = info_dict.copy()
            info_copy['url_list'] = fragment_urls
            info_copy['key_list'] = key_list
            fd = real_downloader(self.ydl, self.params)
            # TODO: Make progress updates work without hooking twice
            # for ph in self._progress_hooks:
            #     fd.add_progress_hook(ph)
            success = fd.real_download(filename, info_copy)
            if not success:
                return False
        else:
            self._finish_frag_download(ctx)
        return True
@@ -27,6 +27,7 @@ from ..utils import (
 class HttpFD(FileDownloader):
    def real_download(self, filename, info_dict):
        url = info_dict['url']
        request_data = info_dict.get('request_data', None)
        class DownloadContext(dict):
            __getattr__ = dict.get
@@ -101,7 +102,7 @@ class HttpFD(FileDownloader):
                range_end = ctx.data_len - 1
            has_range = range_start is not None
            ctx.has_range = has_range
-            request = sanitized_Request(url, None, headers)
+            request = sanitized_Request(url, request_data, headers)
            if has_range:
                set_range(request, range_start, range_end)
            # Establish connection
@@ -152,7 +153,7 @@ class HttpFD(FileDownloader):
                    try:
                        # Open the connection again without the range header
                        ctx.data = self.ydl.urlopen(
-                            sanitized_Request(url, None, headers))
+                            sanitized_Request(url, request_data, headers))
                        content_length = ctx.data.info()['Content-Length']
                    except (compat_urllib_error.HTTPError, ) as err:
                        if err.code < 500 or err.code >= 600:
@@ -0,0 +1,54 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import threading
 from .common import FileDownloader
 from ..downloader import _get_real_downloader
 from ..extractor.niconico import NiconicoIE
 from ..compat import compat_urllib_request
 class NiconicoDmcFD(FileDownloader):
    """ Downloading niconico douga from DMC with heartbeat """
    FD_NAME = 'niconico_dmc'
    def real_download(self, filename, info_dict):
        self.to_screen('[%s] Downloading from DMC' % self.FD_NAME)
        ie = NiconicoIE(self.ydl)
        info_dict, heartbeat_info_dict = ie._get_heartbeat_info(info_dict)
        fd = _get_real_downloader(info_dict, params=self.params)(self.ydl, self.params)
        success = download_complete = False
        timer = [None]
        heartbeat_lock = threading.Lock()
        heartbeat_url = heartbeat_info_dict['url']
        heartbeat_data = heartbeat_info_dict['data']
        heartbeat_interval = heartbeat_info_dict.get('interval', 30)
        self.to_screen('[%s] Heartbeat with %s second interval...' % (self.FD_NAME, heartbeat_interval))
        def heartbeat():
            try:
                compat_urllib_request.urlopen(url=heartbeat_url, data=heartbeat_data.encode())
            except Exception:
                self.to_screen('[%s] Heartbeat failed' % self.FD_NAME)
            with heartbeat_lock:
                if not download_complete:
                    timer[0] = threading.Timer(heartbeat_interval, heartbeat)
                    timer[0].start()
        try:
            heartbeat()
            success = fd.real_download(filename, info_dict)
        finally:
            if heartbeat_lock:
                with heartbeat_lock:
                    timer[0].cancel()
                    download_complete = True
            return success
@@ -1,11 +1,13 @@
 from __future__ import division, unicode_literals
 import re
 import json
 from .fragment import FragmentFD
 from ..compat import compat_urllib_error
-from ..utils import try_get
+from ..utils import (
    try_get,
    RegexNotFoundError,
 )
 from ..extractor.youtube import YoutubeBaseInfoExtractor as YT_BaseIE
@@ -27,31 +29,28 @@ class YoutubeLiveChatReplayFD(FragmentFD):
            'total_frags': None,
        }
-        def dl_fragment(url):
+        ie = YT_BaseIE(self.ydl)
            headers = info_dict.get('http_headers', {})
            return self._download_fragment(ctx, url, info_dict, headers)
-        def parse_yt_initial_data(data):
+        def dl_fragment(url, data=None, headers=None):
-            patterns = (
+            http_headers = info_dict.get('http_headers', {})
-                r'%s\\s*%s' % (YT_BaseIE._YT_INITIAL_DATA_RE, YT_BaseIE._YT_INITIAL_BOUNDARY_RE),
+            if headers:
-                r'%s' % YT_BaseIE._YT_INITIAL_DATA_RE)
+                http_headers = http_headers.copy()
-            data = data.decode('utf-8', 'replace')
+                http_headers.update(headers)
-            for patt in patterns:
+            return self._download_fragment(ctx, url, info_dict, http_headers, data)
                try:
                    raw_json = re.search(patt, data).group(1)
                    return json.loads(raw_json)
                except AttributeError:
                    continue
-        def download_and_parse_fragment(url, frag_index):
+        def download_and_parse_fragment(url, frag_index, request_data):
            count = 0
            while count <= fragment_retries:
                try:
-                    success, raw_fragment = dl_fragment(url)
+                    success, raw_fragment = dl_fragment(url, request_data, {'content-type': 'application/json'})
                    if not success:
                        return False, None, None
-                    data = parse_yt_initial_data(raw_fragment) or json.loads(raw_fragment)['response']
+                    try:
-
+                        data = ie._extract_yt_initial_data(video_id, raw_fragment.decode('utf-8', 'replace'))
                    except RegexNotFoundError:
                        data = None
                    if not data:
                        data = json.loads(raw_fragment)
                    live_chat_continuation = try_get(
                        data,
                        lambda x: x['continuationContents']['liveChatContinuation'], dict) or {}
@@ -84,22 +83,37 @@ class YoutubeLiveChatReplayFD(FragmentFD):
            'https://www.youtube.com/watch?v={}'.format(video_id))
        if not success:
            return False
-        data = parse_yt_initial_data(raw_fragment)
+        try:
            data = ie._extract_yt_initial_data(video_id, raw_fragment.decode('utf-8', 'replace'))
        except RegexNotFoundError:
            return False
        continuation_id = try_get(
            data,
            lambda x: x['contents']['twoColumnWatchNextResults']['conversationBar']['liveChatRenderer']['continuations'][0]['reloadContinuationData']['continuation'])
        # no data yet but required to call _append_fragment
        self._append_fragment(ctx, b'')
        ytcfg = ie._extract_ytcfg(video_id, raw_fragment.decode('utf-8', 'replace'))
        if not ytcfg:
            return False
        api_key = try_get(ytcfg, lambda x: x['INNERTUBE_API_KEY'])
        innertube_context = try_get(ytcfg, lambda x: x['INNERTUBE_CONTEXT'])
        if not api_key or not innertube_context:
            return False
        url = 'https://www.youtube.com/youtubei/v1/live_chat/get_live_chat_replay?key=' + api_key
        frag_index = offset = 0
        while continuation_id is not None:
            frag_index += 1
-            url = ''.join((
+            request_data = {
-                'https://www.youtube.com/live_chat_replay',
+                'context': innertube_context,
-                '/get_live_chat_replay' if frag_index > 1 else '',
+                'continuation': continuation_id,
-                '?continuation=%s' % continuation_id,
+            }
-                '&playerOffsetMs=%d&hidden=false&pbj=1' % max(offset - 5000, 0) if frag_index > 1 else ''))
+            if frag_index > 1:
-            success, continuation_id, offset = download_and_parse_fragment(url, frag_index)
+                request_data['currentPlayerState'] = {'playerOffsetMs': str(max(offset - 5000, 0))}
            success, continuation_id, offset = download_and_parse_fragment(
                url, frag_index, json.dumps(request_data, ensure_ascii=False).encode('utf-8') + b'\n')
            if not success:
                return False
            if test:
@@ -478,11 +478,12 @@ class BrightcoveNewIE(AdobePassIE):
            container = source.get('container')
            ext = mimetype2ext(source.get('type'))
            src = source.get('src')
            skip_unplayable = not self._downloader.params.get('allow_unplayable_formats')
            # https://support.brightcove.com/playback-api-video-fields-reference#key_systems_object
-            if container == 'WVM' or source.get('key_systems'):
+            if skip_unplayable and (container == 'WVM' or source.get('key_systems')):
                num_drm_sources += 1
                continue
-            elif ext == 'ism':
+            elif ext == 'ism' and skip_unplayable:
                continue
            elif ext == 'm3u8' or container == 'M2TS':
                if not src:
@@ -546,7 +547,8 @@ class BrightcoveNewIE(AdobePassIE):
                error = errors[0]
                raise ExtractorError(
                    error.get('message') or error.get('error_subcode') or error['error_code'], expected=True)
-            if sources and num_drm_sources == len(sources):
+            if (not self._downloader.params.get('allow_unplayable_formats')
                    and sources and num_drm_sources == len(sources)):
                raise ExtractorError('This video is DRM protected.', expected=True)
        self._sort_formats(formats)
@@ -95,6 +95,9 @@ class CDAIE(InfoExtractor):
        if 'Ten film jest dostępny dla użytkowników premium' in webpage:
            raise ExtractorError('This video is only available for premium users.', expected=True)
        if re.search(r'niedostępn[ey] w(?:&nbsp;|\s+)Twoim kraju\s*<', webpage):
            self.raise_geo_restricted()
        need_confirm_age = False
        if self._html_search_regex(r'(<form[^>]+action="[^"]*/a/validatebirth[^"]*")',
                                   webpage, 'birthday validate form', default=None):
@@ -147,7 +147,8 @@ class CeskaTelevizeIE(InfoExtractor):
                is_live = item.get('type') == 'LIVE'
                formats = []
                for format_id, stream_url in item.get('streamUrls', {}).items():
-                    if 'drmOnly=true' in stream_url:
+                    if (not self._downloader.params.get('allow_unplayable_formats')
                            and 'drmOnly=true' in stream_url):
                        continue
                    if 'playerType=flash' in stream_url:
                        stream_formats = self._extract_m3u8_formats(
@@ -975,7 +975,7 @@ class InfoExtractor(object):
            video_info['id'] = playlist_id
        if playlist_title:
            video_info['title'] = playlist_title
-        if playlist_description:
+        if playlist_description is not None:
            video_info['description'] = playlist_description
        return video_info
@@ -2358,6 +2358,8 @@ class InfoExtractor(object):
                        extract_Initialization(segment_template)
            return ms_info
        skip_unplayable = not self._downloader.params.get('allow_unplayable_formats')
        mpd_duration = parse_duration(mpd_doc.get('mediaPresentationDuration'))
        formats = []
        for period in mpd_doc.findall(_add_ns('Period')):
@@ -2367,11 +2369,11 @@ class InfoExtractor(object):
                'timescale': 1,
            })
            for adaptation_set in period.findall(_add_ns('AdaptationSet')):
-                if is_drm_protected(adaptation_set):
+                if skip_unplayable and is_drm_protected(adaptation_set):
                    continue
                adaption_set_ms_info = extract_multisegment_info(adaptation_set, period_ms_info)
                for representation in adaptation_set.findall(_add_ns('Representation')):
-                    if is_drm_protected(representation):
+                    if skip_unplayable and is_drm_protected(representation):
                        continue
                    representation_attrib = adaptation_set.attrib.copy()
                    representation_attrib.update(representation.attrib)
@@ -2585,7 +2587,10 @@ class InfoExtractor(object):
         1. [MS-SSTR]: Smooth Streaming Protocol,
            https://msdn.microsoft.com/en-us/library/ff469518.aspx
        """
-        if ism_doc.get('IsLive') == 'TRUE' or ism_doc.find('Protection') is not None:
+        if ism_doc.get('IsLive') == 'TRUE':
            return []
        if (not self._downloader.params.get('allow_unplayable_formats')
                and ism_doc.find('Protection') is not None):
            return []
        duration = int(ism_doc.attrib['Duration'])
@@ -103,7 +103,7 @@ class CrackleIE(InfoExtractor):
            formats = []
            for e in media['MediaURLs']:
-                if e.get('UseDRM') is True:
+                if not self._downloader.params.get('allow_unplayable_formats') and e.get('UseDRM') is True:
                    continue
                format_url = url_or_none(e.get('Path'))
                if not format_url:
@@ -502,8 +502,8 @@ from .hungama import (
 from .hypem import HypemIE
 from .ign import (
    IGNIE,
-    OneUPIE,
+    IGNVideoIE,
-    PCMagIE,
+    IGNArticleIE,
 )
 from .iheart import (
    IHeartRadioIE,
@@ -130,6 +130,7 @@ from .kinja import KinjaEmbedIE
 from .gedi import GediEmbedsIE
 from .rcs import RCSEmbedsIE
 from .bitchute import BitChuteIE
 from .rumble import RumbleEmbedIE
 from .arcpublishing import ArcPublishingIE
 from .medialaan import MedialaanIE
@@ -3338,6 +3339,13 @@ class GenericIE(InfoExtractor):
            return self.playlist_from_matches(
                bitchute_urls, video_id, video_title, ie=BitChuteIE.ie_key())
        rumble_urls = RumbleEmbedIE._extract_urls(webpage)
        if len(rumble_urls) == 1:
            return self.url_result(rumble_urls[0], RumbleEmbedIE.ie_key())
        if rumble_urls:
            return self.playlist_from_matches(
                rumble_urls, video_id, video_title, ie=RumbleEmbedIE.ie_key())
        # Look for HTML5 media
        entries = self._parse_html5_media_entries(url, webpage, video_id, m3u8_id='hls')
        if entries:
@@ -96,7 +96,7 @@ class GloboIE(InfoExtractor):
        video = self._download_json(
            'http://api.globovideos.com/videos/%s/playlist' % video_id,
            video_id)['videos'][0]
-        if video.get('encrypted') is True:
+        if not self._downloader.params.get('allow_unplayable_formats') and video.get('encrypted') is True:
            raise ExtractorError('This video is DRM protected.', expected=True)
        title = video['title']
@@ -141,7 +141,7 @@ class HotStarIE(HotStarBaseIE):
        title = video_data['title']
-        if video_data.get('drmProtected'):
+        if not self._downloader.params.get('allow_unplayable_formats') and video_data.get('drmProtected'):
            raise ExtractorError('This video is DRM protected.', expected=True)
        headers = {'Referer': url}
@@ -3,230 +3,255 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import (
    compat_parse_qs,
    compat_urllib_parse_urlparse,
 )
 from ..utils import (
    HEADRequest,
    determine_ext,
    int_or_none,
    parse_iso8601,
    strip_or_none,
    try_get,
 )
-class IGNIE(InfoExtractor):
+class IGNBaseIE(InfoExtractor):
    def _call_api(self, slug):
        return self._download_json(
            'http://apis.ign.com/{0}/v3/{0}s/slug/{1}'.format(self._PAGE_TYPE, slug), slug)
 class IGNIE(IGNBaseIE):
    """
    Extractor for some of the IGN sites, like www.ign.com, es.ign.com de.ign.com.
    Some videos of it.ign.com are also supported
    """
-    _VALID_URL = r'https?://.+?\.ign\.com/(?:[^/]+/)?(?P<type>videos|show_videos|articles|feature|(?:[^/]+/\d+/video))(/.+)?/(?P<name_or_id>.+)'
+    _VALID_URL = r'https?://(?:.+?\.ign|www\.pcmag)\.com/videos/(?:\d{4}/\d{2}/\d{2}/)?(?P<id>[^/?&#]+)'
    IE_NAME = 'ign.com'
    _PAGE_TYPE = 'video'
-    _API_URL_TEMPLATE = 'http://apis.ign.com/video/v3/videos/%s'
+    _TESTS = [{
-    _EMBED_RE = r'<iframe[^>]+?["\']((?:https?:)?//.+?\.ign\.com.+?/embed.+?)["\']'
+        'url': 'http://www.ign.com/videos/2013/06/05/the-last-of-us-review',
-
+        'md5': 'd2e1586d9987d40fad7867bf96a018ea',
-    _TESTS = [
+        'info_dict': {
-        {
+            'id': '8f862beef863986b2785559b9e1aa599',
-            'url': 'http://www.ign.com/videos/2013/06/05/the-last-of-us-review',
+            'ext': 'mp4',
-            'md5': 'febda82c4bafecd2d44b6e1a18a595f8',
+            'title': 'The Last of Us Review',
-            'info_dict': {
+            'description': 'md5:c8946d4260a4d43a00d5ae8ed998870c',
-                'id': '8f862beef863986b2785559b9e1aa599',
+            'timestamp': 1370440800,
-                'ext': 'mp4',
+            'upload_date': '20130605',
-                'title': 'The Last of Us Review',
+            'tags': 'count:9',
-                'description': 'md5:c8946d4260a4d43a00d5ae8ed998870c',
+        }
-                'timestamp': 1370440800,
+    }, {
-                'upload_date': '20130605',
+        'url': 'http://www.pcmag.com/videos/2015/01/06/010615-whats-new-now-is-gogo-snooping-on-your-data',
-                'uploader_id': 'cberidon@ign.com',
+        'md5': 'f1581a6fe8c5121be5b807684aeac3f6',
-            }
+        'info_dict': {
-        },
+            'id': 'ee10d774b508c9b8ec07e763b9125b91',
-        {
+            'ext': 'mp4',
-            'url': 'http://me.ign.com/en/feature/15775/100-little-things-in-gta-5-that-will-blow-your-mind',
+            'title': 'What\'s New Now: Is GoGo Snooping on Your Data?',
-            'info_dict': {
+            'description': 'md5:817a20299de610bd56f13175386da6fa',
-                'id': '100-little-things-in-gta-5-that-will-blow-your-mind',
+            'timestamp': 1420571160,
-            },
+            'upload_date': '20150106',
-            'playlist': [
+            'tags': 'count:4',
-                {
+        }
-                    'info_dict': {
+    }, {
-                        'id': '5ebbd138523268b93c9141af17bec937',
+        'url': 'https://www.ign.com/videos/is-a-resident-evil-4-remake-on-the-way-ign-daily-fix',
-                        'ext': 'mp4',
+        'only_matching': True,
-                        'title': 'GTA 5 Video Review',
+    }]
                        'description': 'Rockstar drops the mic on this generation of games. Watch our review of the masterly Grand Theft Auto V.',
                        'timestamp': 1379339880,
                        'upload_date': '20130916',
                        'uploader_id': 'danieljkrupa@gmail.com',
                    },
                },
                {
                    'info_dict': {
                        'id': '638672ee848ae4ff108df2a296418ee2',
                        'ext': 'mp4',
                        'title': '26 Twisted Moments from GTA 5 in Slow Motion',
                        'description': 'The twisted beauty of GTA 5 in stunning slow motion.',
                        'timestamp': 1386878820,
                        'upload_date': '20131212',
                        'uploader_id': 'togilvie@ign.com',
                    },
                },
            ],
            'params': {
                'skip_download': True,
            },
        },
        {
            'url': 'http://www.ign.com/articles/2014/08/15/rewind-theater-wild-trailer-gamescom-2014?watch',
            'md5': '618fedb9c901fd086f6f093564ef8558',
            'info_dict': {
                'id': '078fdd005f6d3c02f63d795faa1b984f',
                'ext': 'mp4',
                'title': 'Rewind Theater - Wild Trailer Gamescom 2014',
                'description': 'Brian and Jared explore Michel Ancel\'s captivating new preview.',
                'timestamp': 1408047180,
                'upload_date': '20140814',
                'uploader_id': 'jamesduggan1990@gmail.com',
            },
        },
        {
            'url': 'http://me.ign.com/en/videos/112203/video/how-hitman-aims-to-be-different-than-every-other-s',
            'only_matching': True,
        },
        {
            'url': 'http://me.ign.com/ar/angry-birds-2/106533/video/lrd-ldyy-lwl-lfylm-angry-birds',
            'only_matching': True,
        },
        {
            # videoId pattern
            'url': 'http://www.ign.com/articles/2017/06/08/new-ducktales-short-donalds-birthday-doesnt-go-as-planned',
            'only_matching': True,
        },
    ]
    def _find_video_id(self, webpage):
        res_id = [
            r'"video_id"\s*:\s*"(.*?)"',
            r'class="hero-poster[^"]*?"[^>]*id="(.+?)"',
            r'data-video-id="(.+?)"',
            r'<object id="vid_(.+?)"',
            r'<meta name="og:image" content=".*/(.+?)-(.+?)/.+.jpg"',
            r'videoId&quot;\s*:\s*&quot;(.+?)&quot;',
            r'videoId["\']\s*:\s*["\']([^"\']+?)["\']',
        ]
        return self._search_regex(res_id, webpage, 'video id', default=None)
    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        display_id = self._match_id(url)
-        name_or_id = mobj.group('name_or_id')
+        video = self._call_api(display_id)
-        page_type = mobj.group('type')
+        video_id = video['videoId']
-        webpage = self._download_webpage(url, name_or_id)
+        metadata = video['metadata']
-        if page_type != 'video':
+        title = metadata.get('longTitle') or metadata.get('title') or metadata['name']
            multiple_urls = re.findall(
                r'<param name="flashvars"[^>]*value="[^"]*?url=(https?://www\.ign\.com/videos/.*?)["&]',
                webpage)
            if multiple_urls:
                entries = [self.url_result(u, ie='IGN') for u in multiple_urls]
                return {
                    '_type': 'playlist',
                    'id': name_or_id,
                    'entries': entries,
                }
        video_id = self._find_video_id(webpage)
        if not video_id:
            return self.url_result(self._search_regex(
                self._EMBED_RE, webpage, 'embed url'))
        return self._get_video_info(video_id)
    def _get_video_info(self, video_id):
        api_data = self._download_json(
            self._API_URL_TEMPLATE % video_id, video_id)
        formats = []
-        m3u8_url = api_data['refs'].get('m3uUrl')
+        refs = video.get('refs') or {}
        m3u8_url = refs.get('m3uUrl')
        if m3u8_url:
            formats.extend(self._extract_m3u8_formats(
                m3u8_url, video_id, 'mp4', 'm3u8_native',
                m3u8_id='hls', fatal=False))
-        f4m_url = api_data['refs'].get('f4mUrl')
+
        f4m_url = refs.get('f4mUrl')
        if f4m_url:
            formats.extend(self._extract_f4m_formats(
                f4m_url, video_id, f4m_id='hds', fatal=False))
-        for asset in api_data['assets']:
+
        for asset in (video.get('assets') or []):
            asset_url = asset.get('url')
            if not asset_url:
                continue
            formats.append({
-                'url': asset['url'],
+                'url': asset_url,
-                'tbr': asset.get('actual_bitrate_kbps'),
+                'tbr': int_or_none(asset.get('bitrate'), 1000),
-                'fps': asset.get('frame_rate'),
+                'fps': int_or_none(asset.get('frame_rate')),
                'height': int_or_none(asset.get('height')),
                'width': int_or_none(asset.get('width')),
            })
        mezzanine_url = try_get(video, lambda x: x['system']['mezzanineUrl'])
        if mezzanine_url:
            formats.append({
                'ext': determine_ext(mezzanine_url, 'mp4'),
                'format_id': 'mezzanine',
                'preference': 1,
                'url': mezzanine_url,
            })
        self._sort_formats(formats)
-        thumbnails = [{
+        thumbnails = []
-            'url': thumbnail['url']
+        for thumbnail in (video.get('thumbnails') or []):
-        } for thumbnail in api_data.get('thumbnails', [])]
+            thumbnail_url = thumbnail.get('url')
            if not thumbnail_url:
                continue
            thumbnails.append({
                'url': thumbnail_url,
            })
-        metadata = api_data['metadata']
+        tags = []
        for tag in (video.get('tags') or []):
            display_name = tag.get('displayName')
            if not display_name:
                continue
            tags.append(display_name)
        return {
-            'id': api_data.get('videoId') or video_id,
+            'id': video_id,
-            'title': metadata.get('longTitle') or metadata.get('name') or metadata.get['title'],
+            'title': title,
-            'description': metadata.get('description'),
+            'description': strip_or_none(metadata.get('description')),
            'timestamp': parse_iso8601(metadata.get('publishDate')),
            'duration': int_or_none(metadata.get('duration')),
-            'display_id': metadata.get('slug') or video_id,
+            'display_id': display_id,
            'uploader_id': metadata.get('creator'),
            'thumbnails': thumbnails,
            'formats': formats,
            'tags': tags,
        }
-class OneUPIE(IGNIE):
+class IGNVideoIE(InfoExtractor):
-    _VALID_URL = r'https?://gamevideos\.1up\.com/(?P<type>video)/id/(?P<name_or_id>.+)\.html'
+    _VALID_URL = r'https?://.+?\.ign\.com/(?:[a-z]{2}/)?[^/]+/(?P<id>\d+)/(?:video|trailer)/'
    IE_NAME = '1up.com'
    _TESTS = [{
-        'url': 'http://gamevideos.1up.com/video/id/34976.html',
+        'url': 'http://me.ign.com/en/videos/112203/video/how-hitman-aims-to-be-different-than-every-other-s',
-        'md5': 'c9cc69e07acb675c31a16719f909e347',
+        'md5': 'dd9aca7ed2657c4e118d8b261e5e9de1',
        'info_dict': {
-            'id': '34976',
+            'id': 'e9be7ea899a9bbfc0674accc22a36cc8',
            'ext': 'mp4',
-            'title': 'Sniper Elite V2 - Trailer',
+            'title': 'How Hitman Aims to Be Different Than Every Other Stealth Game - NYCC 2015',
-            'description': 'md5:bf0516c5ee32a3217aa703e9b1bc7826',
+            'description': 'Taking out assassination targets in Hitman has never been more stylish.',
-            'timestamp': 1313099220,
+            'timestamp': 1444665600,
-            'upload_date': '20110811',
+            'upload_date': '20151012',
            'uploader_id': 'IGN',
        }
    }, {
        'url': 'http://me.ign.com/ar/angry-birds-2/106533/video/lrd-ldyy-lwl-lfylm-angry-birds',
        'only_matching': True,
    }, {
        # Youtube embed
        'url': 'https://me.ign.com/ar/ratchet-clank-rift-apart/144327/trailer/embed',
        'only_matching': True,
    }, {
        # Twitter embed
        'url': 'http://adria.ign.com/sherlock-season-4/9687/trailer/embed',
        'only_matching': True,
    }, {
        # Vimeo embed
        'url': 'https://kr.ign.com/bic-2018/3307/trailer/embed',
        'only_matching': True,
    }]
    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        video_id = self._match_id(url)
-        result = super(OneUPIE, self)._real_extract(url)
+        req = HEADRequest(url.rsplit('/', 1)[0] + '/embed')
-        result['id'] = mobj.group('name_or_id')
+        url = self._request_webpage(req, video_id).geturl()
-        return result
+        ign_url = compat_parse_qs(
            compat_urllib_parse_urlparse(url).query).get('url', [None])[0]
        if ign_url:
            return self.url_result(ign_url, IGNIE.ie_key())
        return self.url_result(url)
-class PCMagIE(IGNIE):
+class IGNArticleIE(IGNBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?pcmag\.com/(?P<type>videos|article2)(/.+)?/(?P<name_or_id>.+)'
+    _VALID_URL = r'https?://.+?\.ign\.com/(?:articles(?:/\d{4}/\d{2}/\d{2})?|(?:[a-z]{2}/)?feature/\d+)/(?P<id>[^/?&#]+)'
-    IE_NAME = 'pcmag'
+    _PAGE_TYPE = 'article'
    _EMBED_RE = r'iframe\.setAttribute\("src",\s*__util.objToUrlString\("http://widgets\.ign\.com/video/embed/content\.html?[^"]*url=([^"]+)["&]'
    _TESTS = [{
-        'url': 'http://www.pcmag.com/videos/2015/01/06/010615-whats-new-now-is-gogo-snooping-on-your-data',
+        'url': 'http://me.ign.com/en/feature/15775/100-little-things-in-gta-5-that-will-blow-your-mind',
        'md5': '212d6154fd0361a2781075f1febbe9ad',
        'info_dict': {
-            'id': 'ee10d774b508c9b8ec07e763b9125b91',
+            'id': '524497489e4e8ff5848ece34',
-            'ext': 'mp4',
+            'title': '100 Little Things in GTA 5 That Will Blow Your Mind',
-            'title': '010615_What\'s New Now: Is GoGo Snooping on Your Data?',
+        },
-            'description': 'md5:a7071ae64d2f68cc821c729d4ded6bb3',
+        'playlist': [
-            'timestamp': 1420571160,
+            {
-            'upload_date': '20150106',
+                'info_dict': {
-            'uploader_id': 'cozzipix@gmail.com',
+                    'id': '5ebbd138523268b93c9141af17bec937',
-        }
+                    'ext': 'mp4',
                    'title': 'GTA 5 Video Review',
                    'description': 'Rockstar drops the mic on this generation of games. Watch our review of the masterly Grand Theft Auto V.',
                    'timestamp': 1379339880,
                    'upload_date': '20130916',
                },
            },
            {
                'info_dict': {
                    'id': '638672ee848ae4ff108df2a296418ee2',
                    'ext': 'mp4',
                    'title': '26 Twisted Moments from GTA 5 in Slow Motion',
                    'description': 'The twisted beauty of GTA 5 in stunning slow motion.',
                    'timestamp': 1386878820,
                    'upload_date': '20131212',
                },
            },
        ],
        'params': {
            'playlist_items': '2-3',
            'skip_download': True,
        },
    }, {
-        'url': 'http://www.pcmag.com/article2/0,2817,2470156,00.asp',
+        'url': 'http://www.ign.com/articles/2014/08/15/rewind-theater-wild-trailer-gamescom-2014?watch',
        'md5': '94130c1ca07ba0adb6088350681f16c1',
        'info_dict': {
-            'id': '042e560ba94823d43afcb12ddf7142ca',
+            'id': '53ee806780a81ec46e0790f8',
-            'ext': 'mp4',
+            'title': 'Rewind Theater - Wild Trailer Gamescom 2014',
-            'title': 'HTC\'s Weird New Re Camera - What\'s New Now',
+        },
-            'description': 'md5:53433c45df96d2ea5d0fda18be2ca908',
+        'playlist_count': 2,
-            'timestamp': 1412953920,
+    }, {
-            'upload_date': '20141010',
+        # videoId pattern
-            'uploader_id': 'chris_snyder@pcmag.com',
+        'url': 'http://www.ign.com/articles/2017/06/08/new-ducktales-short-donalds-birthday-doesnt-go-as-planned',
-        }
+        'only_matching': True,
    }, {
        # Youtube embed
        'url': 'https://www.ign.com/articles/2021-mvp-named-in-puppy-bowl-xvii',
        'only_matching': True,
    }, {
        # IMDB embed
        'url': 'https://www.ign.com/articles/2014/08/07/sons-of-anarchy-final-season-trailer',
        'only_matching': True,
    }, {
        # Facebook embed
        'url': 'https://www.ign.com/articles/2017/09/20/marvels-the-punisher-watch-the-new-trailer-for-the-netflix-series',
        'only_matching': True,
    }, {
        # Brightcove embed
        'url': 'https://www.ign.com/articles/2016/01/16/supergirl-goes-flying-with-martian-manhunter-in-new-clip',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        display_id = self._match_id(url)
        article = self._call_api(display_id)
        def entries():
            media_url = try_get(article, lambda x: x['mediaRelations'][0]['media']['metadata']['url'])
            if media_url:
                yield self.url_result(media_url, IGNIE.ie_key())
            for content in (article.get('content') or []):
                for video_url in re.findall(r'(?:\[(?:ignvideo\s+url|youtube\s+clip_id)|<iframe[^>]+src)="([^"]+)"', content):
                    yield self.url_result(video_url)
        return self.playlist_result(
            entries(), article.get('articleId'),
            strip_or_none(try_get(article, lambda x: x['metadata']['headline'])))
@@ -163,7 +163,10 @@ class IviIE(InfoExtractor):
        for f in result.get('files', []):
            f_url = f.get('url')
            content_format = f.get('content_format')
-            if not f_url or '-MDRM-' in content_format or '-FPS-' in content_format:
+            if not f_url:
                continue
            if (not self._downloader.params.get('allow_unplayable_formats')
                    and ('-MDRM-' in content_format or '-FPS-' in content_format)):
                continue
            formats.append({
                'url': f_url,
@@ -309,7 +309,7 @@ class KalturaIE(InfoExtractor):
            if f.get('fileExt') == 'chun':
                continue
            # DRM-protected video, cannot be decrypted
-            if f.get('fileExt') == 'wvm':
+            if not self._downloader.params.get('allow_unplayable_formats') and f.get('fileExt') == 'wvm':
                continue
            if not f.get('fileExt'):
                # QT indicates QuickTime; some videos have broken fileExt
@@ -96,7 +96,9 @@ class LimelightBaseIE(InfoExtractor):
        urls = []
        for stream in pc_item.get('streams', []):
            stream_url = stream.get('url')
-            if not stream_url or stream.get('drmProtected') or stream_url in urls:
+            if not stream_url or stream_url in urls:
                continue
            if not self._downloader.params.get('allow_unplayable_formats') and stream.get('drmProtected'):
                continue
            urls.append(stream_url)
            ext = determine_ext(stream_url)
@@ -1,25 +1,25 @@
 # coding: utf-8
 from __future__ import unicode_literals
-import datetime
+import re
 import functools
 import json
-import math
+import datetime
 from .common import InfoExtractor
 from ..postprocessor.ffmpeg import FFmpegPostProcessor
 from ..compat import (
    compat_parse_qs,
    compat_urllib_parse_urlparse,
 )
 from ..utils import (
    determine_ext,
    dict_get,
    ExtractorError,
    float_or_none,
    InAdvancePagedList,
    int_or_none,
    float_or_none,
    OnDemandPagedList,
    parse_duration,
    parse_iso8601,
    PostProcessingError,
    remove_start,
    try_get,
    unified_timestamp,
@@ -191,37 +191,87 @@ class NiconicoIE(InfoExtractor):
            self._downloader.report_warning('unable to log in: bad username or password')
        return login_ok
-    def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
+    def _get_heartbeat_info(self, info_dict):
        def yesno(boolean):
            return 'yes' if boolean else 'no'
-        session_api_data = api_data['video']['dmcInfo']['session_api']
+        video_id, video_src_id, audio_src_id = info_dict['url'].split(':')[1].split('/')
        session_api_endpoint = session_api_data['urls'][0]
-        format_id = '-'.join(map(lambda s: remove_start(s['id'], 'archive_'), [video_quality, audio_quality]))
+        # Get video webpage for API data.
        webpage, handle = self._download_webpage_handle(
            'http://www.nicovideo.jp/watch/' + video_id, video_id)
        api_data = self._parse_json(self._html_search_regex(
            'data-api-data="([^"]+)"', webpage,
            'API data', default='{}'), video_id)
        session_api_data = try_get(api_data, lambda x: x['video']['dmcInfo']['session_api'])
        session_api_endpoint = try_get(session_api_data, lambda x: x['urls'][0])
        # ping
        self._download_json(
            'https://nvapi.nicovideo.jp/v1/2ab0cbaa/watch', video_id,
            query={'t': try_get(api_data, lambda x: x['video']['dmcInfo']['tracking_id'])},
            headers={
                'Origin': 'https://www.nicovideo.jp',
                'Referer': 'https://www.nicovideo.jp/watch/' + video_id,
                'X-Frontend-Id': '6',
                'X-Frontend-Version': '0'
            })
        yesno = lambda x: 'yes' if x else 'no'
        # m3u8 (encryption)
        if 'encryption' in try_get(api_data, lambda x: x['video']['dmcInfo']) or {}:
            protocol = 'm3u8'
            session_api_http_parameters = {
                'parameters': {
                    'hls_parameters': {
                        'encryption': {
                            'hls_encryption_v1': {
                                'encrypted_key': try_get(api_data, lambda x: x['video']['dmcInfo']['encryption']['hls_encryption_v1']['encrypted_key']),
                                'key_uri': try_get(api_data, lambda x: x['video']['dmcInfo']['encryption']['hls_encryption_v1']['key_uri'])
                            }
                        },
                        'transfer_preset': '',
                        'use_ssl': yesno(session_api_endpoint['is_ssl']),
                        'use_well_known_port': yesno(session_api_endpoint['is_well_known_port']),
                        'segment_duration': 6000
                    }
                }
            }
        # http
        else:
            protocol = 'http'
            session_api_http_parameters = {
                'parameters': {
                    'http_output_download_parameters': {
                        'use_ssl': yesno(session_api_endpoint['is_ssl']),
                        'use_well_known_port': yesno(session_api_endpoint['is_well_known_port']),
                    }
                }
            }
        session_response = self._download_json(
            session_api_endpoint['url'], video_id,
            query={'_format': 'json'},
            headers={'Content-Type': 'application/json'},
-            note='Downloading JSON metadata for %s' % format_id,
+            note='Downloading JSON metadata for %s' % info_dict['format_id'],
            data=json.dumps({
                'session': {
                    'client_info': {
-                        'player_id': session_api_data['player_id'],
+                        'player_id': session_api_data.get('player_id'),
                    },
                    'content_auth': {
-                        'auth_type': session_api_data['auth_types'][session_api_data['protocols'][0]],
+                        'auth_type': try_get(session_api_data, lambda x: x['auth_types'][session_api_data['protocols'][0]]),
-                        'content_key_timeout': session_api_data['content_key_timeout'],
+                        'content_key_timeout': session_api_data.get('content_key_timeout'),
                        'service_id': 'nicovideo',
-                        'service_user_id': session_api_data['service_user_id']
+                        'service_user_id': session_api_data.get('service_user_id')
                    },
-                    'content_id': session_api_data['content_id'],
+                    'content_id': session_api_data.get('content_id'),
                    'content_src_id_sets': [{
                        'content_src_ids': [{
                            'src_id_to_mux': {
-                                'audio_src_ids': [audio_quality['id']],
+                                'audio_src_ids': [audio_src_id],
-                                'video_src_ids': [video_quality['id']],
+                                'video_src_ids': [video_src_id],
                            }
                        }]
                    }],
@@ -229,52 +279,78 @@ class NiconicoIE(InfoExtractor):
                    'content_uri': '',
                    'keep_method': {
                        'heartbeat': {
-                            'lifetime': session_api_data['heartbeat_lifetime']
+                            'lifetime': session_api_data.get('heartbeat_lifetime')
                        }
                    },
-                    'priority': session_api_data['priority'],
+                    'priority': session_api_data.get('priority'),
                    'protocol': {
                        'name': 'http',
                        'parameters': {
-                            'http_parameters': {
+                            'http_parameters': session_api_http_parameters
                                'parameters': {
                                    'http_output_download_parameters': {
                                        'use_ssl': yesno(session_api_endpoint['is_ssl']),
                                        'use_well_known_port': yesno(session_api_endpoint['is_well_known_port']),
                                    }
                                }
                            }
                        }
                    },
-                    'recipe_id': session_api_data['recipe_id'],
+                    'recipe_id': session_api_data.get('recipe_id'),
                    'session_operation_auth': {
                        'session_operation_auth_by_signature': {
-                            'signature': session_api_data['signature'],
+                            'signature': session_api_data.get('signature'),
-                            'token': session_api_data['token'],
+                            'token': session_api_data.get('token'),
                        }
                    },
                    'timing_constraint': 'unlimited'
                }
            }).encode())
-        resolution = video_quality.get('resolution', {})
+        info_dict['url'] = session_response['data']['session']['content_uri']
        info_dict['protocol'] = protocol
        # get heartbeat info
        heartbeat_info_dict = {
            'url': session_api_endpoint['url'] + '/' + session_response['data']['session']['id'] + '?_format=json&_method=PUT',
            'data': json.dumps(session_response['data']),
            # interval, convert milliseconds to seconds, then halve to make a buffer.
            'interval': float_or_none(session_api_data.get('heartbeat_lifetime'), scale=2000),
        }
        return info_dict, heartbeat_info_dict
    def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
        def parse_format_id(id_code):
            mobj = re.match(r'''(?x)
                    (?:archive_)?
                    (?:(?P<codec>[^_]+)_)?
                    (?:(?P<br>[\d]+)kbps_)?
                    (?:(?P<res>[\d+]+)p_)?
                ''', '%s_' % id_code)
            return mobj.groupdict() if mobj else {}
        protocol = 'niconico_dmc'
        format_id = '-'.join(map(lambda s: remove_start(s['id'], 'archive_'), [video_quality, audio_quality]))
        vdict = parse_format_id(video_quality['id'])
        adict = parse_format_id(audio_quality['id'])
        resolution = video_quality.get('resolution', {'height': vdict.get('res')})
        return {
-            'url': session_response['data']['session']['content_uri'],
+            'url': '%s:%s/%s/%s' % (protocol, video_id, video_quality['id'], audio_quality['id']),
            'format_id': format_id,
            'ext': 'mp4',  # Session API are used in HTML5, which always serves mp4
-            'abr': float_or_none(audio_quality.get('bitrate'), 1000),
+            'vcodec': vdict.get('codec'),
-            'vbr': float_or_none(video_quality.get('bitrate'), 1000),
+            'acodec': adict.get('codec'),
-            'height': resolution.get('height'),
+            'vbr': float_or_none(video_quality.get('bitrate'), 1000) or float_or_none(vdict.get('br')),
-            'width': resolution.get('width'),
+            'abr': float_or_none(audio_quality.get('bitrate'), 1000) or float_or_none(adict.get('br')),
            'height': int_or_none(resolution.get('height', vdict.get('res'))),
            'width': int_or_none(resolution.get('width')),
            'quality': -2 if 'low' in format_id else -1,  # Default quality value is -1
            'protocol': protocol,
            'http_headers': {
                'Origin': 'https://www.nicovideo.jp',
                'Referer': 'https://www.nicovideo.jp/watch/' + video_id,
            }
        }
    def _real_extract(self, url):
        video_id = self._match_id(url)
-        # Get video webpage. We are not actually interested in it for normal
+        # Get video webpage for API data.
        # cases, but need the cookies in order to be able to download the
        # info webpage
        webpage, handle = self._download_webpage_handle(
            'http://www.nicovideo.jp/watch/' + video_id, video_id)
        if video_id.startswith('so'):
@@ -284,80 +360,134 @@ class NiconicoIE(InfoExtractor):
            'data-api-data="([^"]+)"', webpage,
            'API data', default='{}'), video_id)
-        def _format_id_from_url(video_url):
+        def get_video_info_web(items):
-            return 'economy' if video_real_url.endswith('low') else 'normal'
+            return dict_get(api_data['video'], items)
        # Get video info
        video_info_xml = self._download_xml(
            'http://ext.nicovideo.jp/api/getthumbinfo/' + video_id,
            video_id, note='Downloading video info page')
        def get_video_info_xml(items):
            if not isinstance(items, list):
                items = [items]
            for item in items:
                ret = xpath_text(video_info_xml, './/' + item)
                if ret:
                    return ret
        if get_video_info_xml('error'):
            error_code = get_video_info_xml('code')
            if error_code == 'DELETED':
                raise ExtractorError('The video has been deleted.',
                                     expected=True)
            elif error_code == 'NOT_FOUND':
                raise ExtractorError('The video is not found.',
                                     expected=True)
            elif error_code == 'COMMUNITY':
                self.to_screen('%s: The video is community members only.' % video_id)
            else:
                raise ExtractorError('%s reports error: %s' % (self.IE_NAME, error_code))
        # Start extracting video formats
        formats = []
        # Get HTML5 videos info
        try:
            dmc_info = api_data['video']['dmcInfo']
        except KeyError:
            raise ExtractorError('The video can\'t downloaded.',
                                 expected=True)
        quality_info = dmc_info.get('quality')
        for audio_quality in quality_info.get('audios') or {}:
            for video_quality in quality_info.get('videos') or {}:
                if not audio_quality.get('available') or not video_quality.get('available'):
                    continue
                formats.append(self._extract_format_for_quality(
                    api_data, video_id, audio_quality, video_quality))
        # Get flv/swf info
        video_real_url = try_get(api_data, lambda x: x['video']['smileInfo']['url'])
        is_economy = video_real_url.endswith('low')
        if is_economy:
            self.report_warning('Site is currently in economy mode! You will only have access to lower quality streams')
        # Invoking ffprobe to determine resolution
        pp = FFmpegPostProcessor(self._downloader)
        cookies = self._get_cookies('https://nicovideo.jp').output(header='', sep='; path=/; domain=nicovideo.jp;\n')
        self.to_screen('%s: %s' % (video_id, 'Checking smile format with ffprobe'))
        try:
-            video_real_url = api_data['video']['smileInfo']['url']
+            metadata = pp.get_metadata_object(video_real_url, ['-cookies', cookies])
-        except KeyError:  # Flash videos
+        except PostProcessingError as err:
-            # Get flv info
+            raise ExtractorError(err.msg, expected=True)
            flv_info_webpage = self._download_webpage(
                'http://flapi.nicovideo.jp/api/getflv/' + video_id + '?as3=1',
                video_id, 'Downloading flv info')
-            flv_info = compat_parse_qs(flv_info_webpage)
+        v_stream = a_stream = {}
            if 'url' not in flv_info:
                if 'deleted' in flv_info:
                    raise ExtractorError('The video has been deleted.',
                                         expected=True)
                elif 'closed' in flv_info:
                    raise ExtractorError('Niconico videos now require logging in',
                                         expected=True)
                elif 'error' in flv_info:
                    raise ExtractorError('%s reports error: %s' % (
                        self.IE_NAME, flv_info['error'][0]), expected=True)
                else:
                    raise ExtractorError('Unable to find video URL')
-            video_info_xml = self._download_xml(
+        # Some complex swf files doesn't have video stream (e.g. nm4809023)
-                'http://ext.nicovideo.jp/api/getthumbinfo/' + video_id,
+        for stream in metadata['streams']:
-                video_id, note='Downloading video info page')
+            if stream['codec_type'] == 'video':
                v_stream = stream
            elif stream['codec_type'] == 'audio':
                a_stream = stream
-            def get_video_info(items):
+        # Community restricted videos seem to have issues with the thumb API not returning anything at all
-                if not isinstance(items, list):
+        filesize = int(
-                    items = [items]
+            (get_video_info_xml('size_high') if not is_economy else get_video_info_xml('size_low'))
-                for item in items:
+            or metadata['format']['size']
-                    ret = xpath_text(video_info_xml, './/' + item)
+        )
-                    if ret:
+        extension = (
-                        return ret
+            get_video_info_xml('movie_type')
            or 'mp4' if 'mp4' in metadata['format']['format_name'] else metadata['format']['format_name']
        )
-            video_real_url = flv_info['url'][0]
+        # 'creation_time' tag on video stream of re-encoded SMILEVIDEO mp4 files are '1970-01-01T00:00:00.000000Z'.
        timestamp = (
            parse_iso8601(get_video_info_web('first_retrieve'))
            or unified_timestamp(get_video_info_web('postedDateTime'))
        )
        metadata_timestamp = (
            parse_iso8601(try_get(v_stream, lambda x: x['tags']['creation_time']))
            or timestamp if extension != 'mp4' else 0
        )
-            extension = get_video_info('movie_type')
+        # According to compconf, smile videos from pre-2017 are always better quality than their DMC counterparts
-            if not extension:
+        smile_threshold_timestamp = parse_iso8601('2016-12-08T00:00:00+09:00')
                extension = determine_ext(video_real_url)
-            formats = [{
+        is_source = timestamp < smile_threshold_timestamp or metadata_timestamp > 0
        # If movie file size is unstable, old server movie is not source movie.
        if filesize > 1:
            formats.append({
                'url': video_real_url,
                'format_id': 'smile' if not is_economy else 'smile_low',
                'format_note': 'SMILEVIDEO source' if not is_economy else 'SMILEVIDEO low quality',
                'ext': extension,
-                'format_id': _format_id_from_url(video_real_url),
+                'container': extension,
-            }]
+                'vcodec': v_stream.get('codec_name'),
-        else:
+                'acodec': a_stream.get('codec_name'),
-            formats = []
+                # Some complex swf files doesn't have total bit rate metadata (e.g. nm6049209)
                'tbr': int_or_none(metadata['format'].get('bit_rate'), scale=1000),
                'vbr': int_or_none(v_stream.get('bit_rate'), scale=1000),
                'abr': int_or_none(a_stream.get('bit_rate'), scale=1000),
                'height': int_or_none(v_stream.get('height')),
                'width': int_or_none(v_stream.get('width')),
                'source_preference': 5 if not is_economy else -2,
                'quality': 5 if is_source and not is_economy else None,
                'filesize': filesize
            })
-            dmc_info = api_data['video'].get('dmcInfo')
+        if len(formats) == 0:
-            if dmc_info:  # "New" HTML5 videos
+            raise ExtractorError('Unable to find video info.')
                quality_info = dmc_info['quality']
                for audio_quality in quality_info['audios']:
                    for video_quality in quality_info['videos']:
                        if not audio_quality['available'] or not video_quality['available']:
                            continue
                        formats.append(self._extract_format_for_quality(
                            api_data, video_id, audio_quality, video_quality))
-                self._sort_formats(formats)
+        self._sort_formats(formats)
            else:  # "Old" HTML5 videos
                formats = [{
                    'url': video_real_url,
                    'ext': 'mp4',
                    'format_id': _format_id_from_url(video_real_url),
                }]
            def get_video_info(items):
                return dict_get(api_data['video'], items)
        # Start extracting information
-        title = get_video_info('title')
+        title = get_video_info_web('originalTitle')
        if not title:
            title = self._og_search_title(webpage, default=None)
        if not title:
@@ -372,14 +502,13 @@ class NiconicoIE(InfoExtractor):
        video_detail = watch_api_data.get('videoDetail', {})
        thumbnail = (
-            get_video_info(['thumbnail_url', 'thumbnailURL'])
+            self._html_search_regex(r'<meta property="og:image" content="([^"]+)">', webpage, 'thumbnail data', default=None)
            or get_video_info_web(['thumbnail_url', 'largeThumbnailURL', 'thumbnailURL'])
            or self._html_search_meta('image', webpage, 'thumbnail', default=None)
            or video_detail.get('thumbnail'))
-        description = get_video_info('description')
+        description = get_video_info_web('description')
        timestamp = (parse_iso8601(get_video_info('first_retrieve'))
                     or unified_timestamp(get_video_info('postedDateTime')))
        if not timestamp:
            match = self._html_search_meta('datePublished', webpage, 'date published', default=None)
            if match:
@@ -389,7 +518,7 @@ class NiconicoIE(InfoExtractor):
                video_detail['postedAt'].replace('/', '-'),
                delimiter=' ', timezone=datetime.timedelta(hours=9))
-        view_count = int_or_none(get_video_info(['view_counter', 'viewCount']))
+        view_count = int_or_none(get_video_info_web(['view_counter', 'viewCount']))
        if not view_count:
            match = self._html_search_regex(
                r'>Views: <strong[^>]*>([^<]+)</strong>',
@@ -398,7 +527,7 @@ class NiconicoIE(InfoExtractor):
                view_count = int_or_none(match.replace(',', ''))
        view_count = view_count or video_detail.get('viewCount')
-        comment_count = (int_or_none(get_video_info('comment_num'))
+        comment_count = (int_or_none(get_video_info_web('comment_num'))
                         or video_detail.get('commentCount')
                         or try_get(api_data, lambda x: x['thread']['commentCount']))
        if not comment_count:
@@ -409,19 +538,19 @@ class NiconicoIE(InfoExtractor):
                comment_count = int_or_none(match.replace(',', ''))
        duration = (parse_duration(
-            get_video_info('length')
+            get_video_info_web('length')
            or self._html_search_meta(
                'video:duration', webpage, 'video duration', default=None))
            or video_detail.get('length')
-            or get_video_info('duration'))
+            or get_video_info_web('duration'))
-        webpage_url = get_video_info('watch_url') or url
+        webpage_url = get_video_info_web('watch_url') or url
        # Note: cannot use api_data.get('owner', {}) because owner may be set to "null"
        # in the JSON, which will cause None to be returned instead of {}.
        owner = try_get(api_data, lambda x: x.get('owner'), dict) or {}
-        uploader_id = get_video_info(['ch_id', 'user_id']) or owner.get('id')
+        uploader_id = get_video_info_web(['ch_id', 'user_id']) or owner.get('id')
-        uploader = get_video_info(['ch_name', 'user_nickname']) or owner.get('nickname')
+        uploader = get_video_info_web(['ch_name', 'user_nickname']) or owner.get('nickname')
        return {
            'id': video_id,
@@ -456,60 +585,45 @@ class NiconicoPlaylistIE(InfoExtractor):
        'url': 'https://www.nicovideo.jp/user/805442/mylist/27411728',
        'only_matching': True,
    }]
    _PAGE_SIZE = 100
    def _call_api(self, list_id, resource, query):
        return self._download_json(
            'https://nvapi.nicovideo.jp/v2/mylists/' + list_id, list_id,
            'Downloading %s JSON metatdata' % resource, query=query,
            headers={'X-Frontend-Id': 6})['data']['mylist']
    def _parse_owner(self, item):
        owner = item.get('owner') or {}
        if owner:
            return {
                'uploader': owner.get('name'),
                'uploader_id': owner.get('id'),
            }
        return {}
    def _fetch_page(self, list_id, page):
        page += 1
        items = self._call_api(list_id, 'page %d' % page, {
            'page': page,
            'pageSize': self._PAGE_SIZE,
        })['items']
        for item in items:
            video = item.get('video') or {}
            video_id = video.get('id')
            if not video_id:
                continue
            count = video.get('count') or {}
            get_count = lambda x: int_or_none(count.get(x))
            info = {
                '_type': 'url',
                'id': video_id,
                'title': video.get('title'),
                'url': 'https://www.nicovideo.jp/watch/' + video_id,
                'description': video.get('shortDescription'),
                'duration': int_or_none(video.get('duration')),
                'view_count': get_count('view'),
                'comment_count': get_count('comment'),
                'ie_key': NiconicoIE.ie_key(),
            }
            info.update(self._parse_owner(video))
            yield info
    def _real_extract(self, url):
        list_id = self._match_id(url)
-        mylist = self._call_api(list_id, 'list', {
+        webpage = self._download_webpage(url, list_id)
-            'pageSize': 1,
+
-        })
+        header = self._parse_json(self._html_search_regex(
-        entries = InAdvancePagedList(
+            r'data-common-header="([^"]+)"', webpage,
-            functools.partial(self._fetch_page, list_id),
+            'webpage header'), list_id)
-            math.ceil(mylist['totalItemCount'] / self._PAGE_SIZE),
+        frontendId = header.get('initConfig').get('frontendId')
-            self._PAGE_SIZE)
+        frontendVersion = header.get('initConfig').get('frontendVersion')
-        result = self.playlist_result(
+
-            entries, list_id, mylist.get('name'), mylist.get('description'))
+        def get_page_data(pagenum, pagesize):
-        result.update(self._parse_owner(mylist))
+            return self._download_json(
-        return result
+                'http://nvapi.nicovideo.jp/v2/mylists/' + list_id, list_id,
                query={'page': 1 + pagenum, 'pageSize': pagesize},
                headers={
                    'X-Frontend-Id': frontendId,
                    'X-Frontend-Version': frontendVersion,
                }).get('data').get('mylist')
        data = get_page_data(0, 1)
        title = data.get('name')
        description = data.get('description')
        uploader = data.get('owner').get('name')
        uploader_id = data.get('owner').get('id')
        def pagefunc(pagenum):
            data = get_page_data(pagenum, 25)
            return ({
                '_type': 'url',
                'url': 'http://www.nicovideo.jp/watch/' + item.get('watchId'),
            } for item in data.get('items'))
        return {
            '_type': 'playlist',
            'id': list_id,
            'title': title,
            'description': description,
            'uploader': uploader,
            'uploader_id': uploader_id,
            'entries': OnDemandPagedList(pagefunc, 25),
        }
@@ -36,7 +36,8 @@ class NineCNineMediaIE(InfoExtractor):
                '$include': '[HasClosedCaptions]',
            })
-        if try_get(content_package, lambda x: x['Constraints']['Security']['Type']):
+        if (not self._downloader.params.get('allow_unplayable_formats')
                and try_get(content_package, lambda x: x['Constraints']['Security']['Type'])):
            raise ExtractorError('This video is DRM protected.', expected=True)
        manifest_base_url = content_package_url + 'manifest.'
@@ -66,7 +66,7 @@ class NineNowIE(InfoExtractor):
        video_data = common_data['video']
-        if video_data.get('drm'):
+        if not self._downloader.params.get('allow_unplayable_formats') and video_data.get('drm'):
            raise ExtractorError('This video is DRM protected.', expected=True)
        brightcove_id = video_data.get('brightcoveId') or 'ref:' + video_data['referenceId']
@@ -246,7 +246,7 @@ class NPOIE(NPOBaseIE):
                })
        if not formats:
-            if drm:
+            if not self._downloader.params.get('allow_unplayable_formats') and drm:
                raise ExtractorError('This video is DRM protected.', expected=True)
            return
@@ -34,7 +34,7 @@ class ProSiebenSat1BaseIE(InfoExtractor):
                'ids': clip_id,
            })[0]
-        if video.get('is_protected') is True:
+        if not self._downloader.params.get('allow_unplayable_formats') and video.get('is_protected') is True:
            raise ExtractorError('This video is DRM protected.', expected=True)
        formats = []
@@ -125,7 +125,7 @@ class RTBFIE(InfoExtractor):
                })
        mpd_url = data.get('urlDash')
-        if not data.get('drm') and mpd_url:
+        if mpd_url and (self._downloader.params.get('allow_unplayable_formats') or not data.get('drm')):
            formats.extend(self._extract_mpd_formats(
                mpd_url, media_id, mpd_id='dash', fatal=False))
@@ -1,6 +1,8 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
@@ -28,6 +30,14 @@ class RumbleEmbedIE(InfoExtractor):
        'only_matching': True,
    }]
    @staticmethod
    def _extract_urls(webpage):
        return [
            mobj.group('url')
            for mobj in re.finditer(
                r'(?:<(?:script|iframe)[^>]+\bsrc=|["\']embedUrl["\']\s*:\s*)["\'](?P<url>%s)' % RumbleEmbedIE._VALID_URL,
                webpage)]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        video = self._download_json(
@@ -200,8 +200,8 @@ class RuutuIE(InfoExtractor):
                return node.get('value')
        if not formats:
-            drm = xpath_text(video_xml, './Clip/DRM', default=None)
+            if (not self._downloader.params.get('allow_unplayable_formats')
-            if drm:
+                    and xpath_text(video_xml, './Clip/DRM', default=None)):
                raise ExtractorError('This video is DRM protected.', expected=True)
            ns_st_cds = pv('ns_st_cds')
            if ns_st_cds != 'free':
@@ -111,7 +111,7 @@ class ShahidIE(ShahidBaseIE):
        playout = self._call_api(
            'playout/url/' + video_id, video_id)['playout']
-        if playout.get('drm'):
+        if not self._downloader.params.get('allow_unplayable_formats') and playout.get('drm'):
            raise ExtractorError('This video is DRM protected.', expected=True)
        formats = self._extract_m3u8_formats(playout['url'], video_id, 'mp4')
@@ -75,7 +75,7 @@ class SonyLIVIE(InfoExtractor):
        video_id = self._match_id(url)
        content = self._call_api(
            '1.5', 'IN/CONTENT/VIDEOURL/VOD/' + video_id, video_id)
-        if content.get('isEncrypted'):
+        if not self._downloader.params.get('allow_unplayable_formats') and content.get('isEncrypted'):
            raise ExtractorError('This video is DRM protected.', expected=True)
        dash_url = content['videoURL']
        headers = {
@@ -154,7 +154,8 @@ class ToggleIE(InfoExtractor):
                })
        if not formats:
            for meta in (info.get('Metas') or []):
-                if meta.get('Key') == 'Encryption' and meta.get('Value') == '1':
+                if (not self._downloader.params.get('allow_unplayable_formats')
                        and meta.get('Key') == 'Encryption' and meta.get('Value') == '1'):
                    raise ExtractorError(
                        'This video is DRM protected.', expected=True)
            # Most likely because geo-blocked
@@ -74,7 +74,7 @@ class TouTvIE(RadioCanadaIE):
            })
        # IsDrm does not necessarily mean the video is DRM protected (see
        # https://github.com/ytdl-org/youtube-dl/issues/13994).
-        if metadata.get('IsDrm'):
+        if not self._downloader.params.get('allow_unplayable_formats') and metadata.get('IsDrm'):
            self.report_warning('This video is probably DRM protected.', path)
        video_id = metadata['IdMedia']
        details = metadata['Details']
@@ -69,7 +69,7 @@ class TVNowBaseIE(InfoExtractor):
            if formats:
                break
        else:
-            if info.get('isDrm'):
+            if not self._downloader.params.get('allow_unplayable_formats') and info.get('isDrm'):
                raise ExtractorError(
                    'Video %s is DRM protected' % video_id, expected=True)
            if info.get('geoblocked'):
@@ -42,8 +42,8 @@ class URPlayIE(InfoExtractor):
        url = url.replace('skola.se/Produkter', 'play.se/program')
        webpage = self._download_webpage(url, video_id)
        urplayer_data = self._parse_json(self._html_search_regex(
-            r'data-react-class="components/Player/Player"[^>]+data-react-props="({.+?})"',
+            r'data-react-class="routes/Product/components/ProgramContainer/ProgramContainer"[^>]+data-react-props="({.+?})"',
-            webpage, 'urplayer data'), video_id)['currentProduct']
+            webpage, 'urplayer data'), video_id)['accessibleEpisodes'][0]
        episode = urplayer_data['title']
        host = self._download_json('http://streaming-loadbalancer.ur.se/loadbalancer.json', video_id)['redirect']
@@ -315,7 +315,7 @@ class VikiIE(VikiBaseIE):
                # Despite CODECS metadata in m3u8 all video-only formats
                # are actually video+audio
                for f in m3u8_formats:
-                    if '_drm/index_' in f['url']:
+                    if not self._downloader.params.get('allow_unplayable_formats') and '_drm/index_' in f['url']:
                        continue
                    if f.get('acodec') == 'none' and f.get('vcodec') != 'none':
                        f['acodec'] = None
@@ -41,12 +41,13 @@ class WakanimIE(InfoExtractor):
        m3u8_url = urljoin(url, self._search_regex(
            r'file\s*:\s*(["\'])(?P<url>(?:(?!\1).)+)\1', webpage, 'm3u8 url',
            group='url'))
-        # https://docs.microsoft.com/en-us/azure/media-services/previous/media-services-content-protection-overview#streaming-urls
+        if not self._downloader.params.get('allow_unplayable_formats'):
-        encryption = self._search_regex(
+            # https://docs.microsoft.com/en-us/azure/media-services/previous/media-services-content-protection-overview#streaming-urls
-            r'encryption%3D(c(?:enc|bc(?:s-aapl)?))',
+            encryption = self._search_regex(
-            m3u8_url, 'encryption', default=None)
+                r'encryption%3D(c(?:enc|bc(?:s-aapl)?))',
-        if encryption and encryption in ('cenc', 'cbcs-aapl'):
+                m3u8_url, 'encryption', default=None)
-            raise ExtractorError('This video is DRM protected.', expected=True)
+            if encryption in ('cenc', 'cbcs-aapl'):
                raise ExtractorError('This video is DRM protected.', expected=True)
        formats = self._extract_m3u8_formats(
            m3u8_url, video_id, 'mp4', entry_protocol='m3u8_native',
@@ -11,11 +11,14 @@ from ..utils import (
    dict_get,
    extract_attributes,
    ExtractorError,
    float_or_none,
    int_or_none,
    parse_duration,
    str_or_none,
    try_get,
    unified_strdate,
    url_or_none,
    urljoin,
 )
@@ -146,36 +149,89 @@ class XHamsterIE(InfoExtractor):
            video = initials['videoModel']
            title = video['title']
            formats = []
-            for format_id, formats_dict in video['sources'].items():
+            format_urls = set()
            format_sizes = {}
            sources = try_get(video, lambda x: x['sources'], dict) or {}
            for format_id, formats_dict in sources.items():
                if not isinstance(formats_dict, dict):
                    continue
                download_sources = try_get(sources, lambda x: x['download'], dict) or {}
                for quality, format_dict in download_sources.items():
                    if not isinstance(format_dict, dict):
                        continue
                    format_sizes[quality] = float_or_none(format_dict.get('size'))
                for quality, format_item in formats_dict.items():
                    if format_id == 'download':
                        # Download link takes some time to be generated,
                        # skipping for now
                        continue
-                        if not isinstance(format_item, dict):
+                    format_url = format_item
                            continue
                        format_url = format_item.get('link')
                        filesize = int_or_none(
                            format_item.get('size'), invscale=1000000)
                    else:
                        format_url = format_item
                        filesize = None
                    format_url = url_or_none(format_url)
-                    if not format_url:
+                    if not format_url or format_url in format_urls:
                        continue
                    format_urls.add(format_url)
                    formats.append({
                        'format_id': '%s-%s' % (format_id, quality),
                        'url': format_url,
                        'ext': determine_ext(format_url, 'mp4'),
                        'height': get_height(quality),
-                        'filesize': filesize,
+                        'filesize': format_sizes.get(quality),
                        'http_headers': {
                            'Referer': urlh.geturl(),
                        },
                    })
-            self._sort_formats(formats)
+            xplayer_sources = try_get(
                initials, lambda x: x['xplayerSettings']['sources'], dict)
            if xplayer_sources:
                hls_sources = xplayer_sources.get('hls')
                if isinstance(hls_sources, dict):
                    for hls_format_key in ('url', 'fallback'):
                        hls_url = hls_sources.get(hls_format_key)
                        if not hls_url:
                            continue
                        hls_url = urljoin(url, hls_url)
                        if not hls_url or hls_url in format_urls:
                            continue
                        format_urls.add(hls_url)
                        formats.extend(self._extract_m3u8_formats(
                            hls_url, video_id, 'mp4', entry_protocol='m3u8_native',
                            m3u8_id='hls', fatal=False))
                standard_sources = xplayer_sources.get('standard')
                if isinstance(standard_sources, dict):
                    for format_id, formats_list in standard_sources.items():
                        if not isinstance(formats_list, list):
                            continue
                        for standard_format in formats_list:
                            if not isinstance(standard_format, dict):
                                continue
                            for standard_format_key in ('url', 'fallback'):
                                standard_url = standard_format.get(standard_format_key)
                                if not standard_url:
                                    continue
                                standard_url = urljoin(url, standard_url)
                                if not standard_url or standard_url in format_urls:
                                    continue
                                format_urls.add(standard_url)
                                ext = determine_ext(standard_url, 'mp4')
                                if ext == 'm3u8':
                                    formats.extend(self._extract_m3u8_formats(
                                        standard_url, video_id, 'mp4', entry_protocol='m3u8_native',
                                        m3u8_id='hls', fatal=False))
                                    continue
                                quality = (str_or_none(standard_format.get('quality'))
                                           or str_or_none(standard_format.get('label'))
                                           or '')
                                formats.append({
                                    'format_id': '%s-%s' % (format_id, quality),
                                    'url': standard_url,
                                    'ext': ext,
                                    'height': get_height(quality),
                                    'filesize': format_sizes.get(quality),
                                    'http_headers': {
                                        'Referer': standard_url,
                                    },
                                })
            self._sort_formats(formats, field_preference=('height', 'width', 'tbr', 'format_id'))
            categories_list = video.get('categories')
            if isinstance(categories_list, list):
@@ -32,7 +32,7 @@ from ..utils import (
    mimetype2ext,
    parse_codecs,
    parse_duration,
-    # qualities,
+    # qualities,  # TODO: Enable this after fixing formatSort
    remove_start,
    smuggle_url,
    str_or_none,
@@ -414,7 +414,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                     (?(1).+)?                                                # if we found the ID, everything can follow
                     $""" % {'playlist_id': YoutubeBaseInfoExtractor._PLAYLIST_ID_RE}
    _PLAYER_INFO_RE = (
-        r'/(?P<id>[a-zA-Z0-9_-]{8,})/player_ias\.vflset(?:/[a-zA-Z]{2,3}_[a-zA-Z]{2,3})?/base\.js$',
+        r'/s/player/(?P<id>[a-zA-Z0-9_-]{8,})/player',
        r'/(?P<id>[a-zA-Z0-9_-]{8,})/player(?:_ias\.vflset(?:/[a-zA-Z]{2,3}_[a-zA-Z]{2,3})?|-plasma-ias-(?:phone|tablet)-[a-z]{2}_[A-Z]{2}\.vflset)/base\.js$',
        r'\b(?P<id>vfl[a-zA-Z0-9_-]+)\b.*?\.js$',
    )
    _formats = {
@@ -621,6 +622,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                'uploader': 'AfrojackVEVO',
                'uploader_id': 'AfrojackVEVO',
                'upload_date': '20131011',
                'abr': 129.495,
            },
            'params': {
                'youtube_include_dash_manifest': True,
@@ -1134,10 +1136,26 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
            'only_matching': True,
        },
        {
-            # Age-gated video only available with authentication (unavailable
+            # https://github.com/ytdl-org/youtube-dl/pull/28094
-            # via embed page workaround)
+            'url': 'OtqTfy26tG0',
-            'url': 'XgnwCQzjau8',
+            'info_dict': {
-            'only_matching': True,
+                'id': 'OtqTfy26tG0',
                'ext': 'mp4',
                'title': 'Burn Out',
                'description': 'md5:8d07b84dcbcbfb34bc12a56d968b6131',
                'upload_date': '20141120',
                'uploader': 'The Cinematic Orchestra - Topic',
                'uploader_id': 'UCIzsJBIyo8hhpFm1NK0uLgw',
                'uploader_url': r're:https?://(?:www\.)?youtube\.com/channel/UCIzsJBIyo8hhpFm1NK0uLgw',
                'artist': 'The Cinematic Orchestra',
                'track': 'Burn Out',
                'album': 'Every Day',
                'release_data': None,
                'release_year': None,
            },
            'params': {
                'skip_download': True,
            },
        },
    ]
@@ -1230,6 +1248,9 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
        funcname = self._search_regex(
            (r'\b[cs]\s*&&\s*[adf]\.set\([^,]+\s*,\s*encodeURIComponent\s*\(\s*(?P<sig>[a-zA-Z0-9$]+)\(',
             r'\b[a-zA-Z0-9]+\s*&&\s*[a-zA-Z0-9]+\.set\([^,]+\s*,\s*encodeURIComponent\s*\(\s*(?P<sig>[a-zA-Z0-9$]+)\(',
             r'\bm=(?P<sig>[a-zA-Z0-9$]{2})\(decodeURIComponent\(h\.s\)\)',
             r'\bc&&\(c=(?P<sig>[a-zA-Z0-9$]{2})\(decodeURIComponent\(c\)\)',
             r'(?:\b|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2})\s*=\s*function\(\s*a\s*\)\s*{\s*a\s*=\s*a\.split\(\s*""\s*\);[a-zA-Z0-9$]{2}\.[a-zA-Z0-9$]{2}\(a,\d+\)',
             r'(?:\b|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2})\s*=\s*function\(\s*a\s*\)\s*{\s*a\s*=\s*a\.split\(\s*""\s*\)',
             r'(?P<sig>[a-zA-Z0-9$]+)\s*=\s*function\(\s*a\s*\)\s*{\s*a\s*=\s*a\.split\(\s*""\s*\)',
             # Obsolete patterns
@@ -1393,7 +1414,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
        url, smuggled_data = unsmuggle_url(url, {})
        video_id = self._match_id(url)
        base_url = self.http_scheme() + '//www.youtube.com/'
-        webpage_url = base_url + 'watch?v=' + video_id
+        webpage_url = base_url + 'watch?v=' + video_id + '&has_verified=1'
        webpage = self._download_webpage(webpage_url, video_id, fatal=False)
        player_response = None
@@ -1493,7 +1514,9 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
        formats = []
        itags = []
        itag_qualities = {}
        player_url = None
        # TODO: Enable this after fixing formatSort
        # q = qualities(['tiny', 'small', 'medium', 'large', 'hd720', 'hd1080', 'hd1440', 'hd2160', 'hd2880', 'highres'])
        streaming_data = player_response.get('streamingData') or {}
        streaming_formats = streaming_data.get('formats') or []
@@ -1502,6 +1525,16 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
            if fmt.get('targetDurationSec') or fmt.get('drmFamilies'):
                continue
            itag = str_or_none(fmt.get('itag'))
            quality = fmt.get('quality')
            if itag and quality:
                itag_qualities[itag] = quality
            # FORMAT_STREAM_TYPE_OTF(otf=1) requires downloading the init fragment
            # (adding `&sq=0` to the URL) and parsing emsg box to determine the
            # number of fragment that would subsequently requested with (`&sq=N`)
            if fmt.get('type') == 'FORMAT_STREAM_TYPE_OTF':
                continue
            fmt_url = fmt.get('url')
            if not fmt_url:
                sc = compat_parse_qs(fmt.get('signatureCipher'))
@@ -1521,10 +1554,10 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                sp = try_get(sc, lambda x: x['sp'][0]) or 'signature'
                fmt_url += '&' + sp + '=' + signature
            itag = str_or_none(fmt.get('itag'))
            if itag:
                itags.append(itag)
-            quality = fmt.get('quality')
+            tbr = float_or_none(
                fmt.get('averageBitrate') or fmt.get('bitrate'), 1000)
            dct = {
                'asr': int_or_none(fmt.get('audioSampleRate')),
                'filesize': int_or_none(fmt.get('contentLength')),
@@ -1532,9 +1565,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                'format_note': fmt.get('qualityLabel') or quality,
                'fps': int_or_none(fmt.get('fps')),
                'height': int_or_none(fmt.get('height')),
-                # 'quality': q(quality),  # This does not correctly reflect the overall quality of the format
+                # 'quality': q(quality),    # TODO: Enable this after fixing formatSort
-                'tbr': float_or_none(fmt.get(
+                'tbr': tbr,
                    'averageBitrate') or fmt.get('bitrate'), 1000),
                'url': fmt_url,
                'width': fmt.get('width'),
            }
@@ -1545,7 +1577,13 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                if mobj:
                    dct['ext'] = mimetype2ext(mobj.group(1))
                    dct.update(parse_codecs(mobj.group(2)))
-            if dct.get('acodec') == 'none' or dct.get('vcodec') == 'none':
+            no_audio = dct.get('acodec') == 'none'
            no_video = dct.get('vcodec') == 'none'
            if no_audio:
                dct['vbr'] = tbr
            if no_video:
                dct['abr'] = tbr
            if no_audio or no_video:
                dct['downloader_options'] = {
                    # Youtube throttles chunks >~10M
                    'http_chunk_size': 10485760,
@@ -1565,25 +1603,22 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
        if self._downloader.params.get('youtube_include_dash_manifest'):
            dash_manifest_url = streaming_data.get('dashManifestUrl')
            if dash_manifest_url:
                dash_formats = []
                for f in self._extract_mpd_formats(
                        dash_manifest_url, video_id, fatal=False):
                    itag = f['format_id']
                    if itag in itags:
                        continue
                    # if itag in itag_qualities:  # TODO: Enable this after fixing formatSort
                    #     f['quality'] = q(itag_qualities[itag])
                    filesize = int_or_none(self._search_regex(
                        r'/clen/(\d+)', f.get('fragment_base_url')
                        or f['url'], 'file size', default=None))
                    if filesize:
                        f['filesize'] = filesize
-                    dash_formats.append(f)
+                    formats.append(f)
                # Until further investigation prefer DASH formats as non-DASH
                # may not be available (see [1])
                # 1. https://github.com/ytdl-org/youtube-dl/issues/28070
                if dash_formats:
                    dash_formats_keys = [f['format_id'] for f in dash_formats]
                    formats = [f for f in formats if f['format_id'] not in dash_formats_keys]
                    formats.extend(dash_formats)
        if not formats:
-            if streaming_data.get('licenseInfos'):
+            if not self._downloader.params.get('allow_unplayable_formats') and streaming_data.get('licenseInfos'):
                raise ExtractorError(
                    'This video is DRM protected.', expected=True)
            pemr = try_get(
@@ -1747,7 +1782,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                    'artist': mobj.group('clean_artist') or ', '.join(a.strip() for a in mobj.group('artist').split('·')),
                    'track': mobj.group('track').strip(),
                    'release_date': release_date,
-                    'release_year': int(release_year),
+                    'release_year': int_or_none(release_year),
                })
        initial_data = None
@@ -1999,8 +2034,10 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                    raise ExtractorError('Unexpected HTTP error code: %s' % response_code)
            first_continuation = True
            chain_msg = ''
            self.to_screen('Downloading comments')
            while continuations:
-                continuation, itct = continuations.pop()
+                continuation = continuations.pop()
                comment_response = get_continuation(continuation, xsrf_token)
                if not comment_response:
                    continue
@@ -2046,9 +2083,10 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                            continue
                        if self._downloader.params.get('verbose', False):
-                            self.to_screen('[debug] Comments downloaded (chain %s) %s of ~%s' % (comment['commentId'], len(video_comments), expected_video_comment_count))
+                            chain_msg = ' (chain %s)' % comment['commentId']
                        self.to_screen('Comments downloaded: %d of ~%d%s' % (len(video_comments), expected_video_comment_count, chain_msg))
                        reply_comment_meta = replies_data[1]['response']['continuationContents']['commentRepliesContinuation']
-                        for reply_meta in replies_data[1]['response']['continuationContents']['commentRepliesContinuation']['contents']:
+                        for reply_meta in reply_comment_meta.get('contents', {}):
                            reply_comment = reply_meta['commentRenderer']
                            video_comments.append({
                                'id': reply_comment['commentId'],
@@ -2063,12 +2101,12 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                            continue
                        reply_continuations += [rcn['nextContinuationData']['continuation'] for rcn in reply_comment_meta['continuations']]
-                self.to_screen('Comments downloaded %s of ~%s' % (len(video_comments), expected_video_comment_count))
+                self.to_screen('Comments downloaded: %d of ~%d' % (len(video_comments), expected_video_comment_count))
                if 'continuations' in item_section:
                    continuations += [ncd['nextContinuationData']['continuation'] for ncd in item_section['continuations']]
                time.sleep(1)
-            self.to_screen('Total comments downloaded %s of ~%s' % (len(video_comments), expected_video_comment_count))
+            self.to_screen('Total comments downloaded: %d of ~%d' % (len(video_comments), expected_video_comment_count))
            info.update({
                'comments': video_comments,
                'comment_count': expected_video_comment_count
@@ -2108,6 +2146,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            'id': 'UCqj7Cz7revf5maW9g5pgNcg',
            'title': 'Игорь Клейнер - Playlists',
            'description': 'md5:be97ee0f14ee314f1f002cf187166ee2',
            'uploader': 'Игорь Клейнер',
            'uploader_id': 'UCqj7Cz7revf5maW9g5pgNcg',
        },
    }, {
        # playlists, multipage, different order
@@ -2117,6 +2157,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            'id': 'UCqj7Cz7revf5maW9g5pgNcg',
            'title': 'Игорь Клейнер - Playlists',
            'description': 'md5:be97ee0f14ee314f1f002cf187166ee2',
            'uploader_id': 'UCqj7Cz7revf5maW9g5pgNcg',
            'uploader': 'Игорь Клейнер',
        },
    }, {
        # playlists, singlepage
@@ -2126,6 +2168,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            'id': 'UCAEtajcuhQ6an9WEzY9LEMQ',
            'title': 'ThirstForScience - Playlists',
            'description': 'md5:609399d937ea957b0f53cbffb747a14c',
            'uploader': 'ThirstForScience',
            'uploader_id': 'UCAEtajcuhQ6an9WEzY9LEMQ',
        }
    }, {
        'url': 'https://www.youtube.com/c/ChristophLaimer/playlists',
@@ -2157,6 +2201,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
            'title': 'lex will - Home',
            'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
            'uploader': 'lex will',
            'uploader_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
        },
        'playlist_mincount': 2,
    }, {
@@ -2166,6 +2212,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
            'title': 'lex will - Videos',
            'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
            'uploader': 'lex will',
            'uploader_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
        },
        'playlist_mincount': 975,
    }, {
@@ -2175,6 +2223,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
            'title': 'lex will - Videos',
            'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
            'uploader': 'lex will',
            'uploader_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
        },
        'playlist_mincount': 199,
    }, {
@@ -2184,6 +2234,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
            'title': 'lex will - Playlists',
            'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
            'uploader': 'lex will',
            'uploader_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
        },
        'playlist_mincount': 17,
    }, {
@@ -2193,6 +2245,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
            'title': 'lex will - Community',
            'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
            'uploader': 'lex will',
            'uploader_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
        },
        'playlist_mincount': 18,
    }, {
@@ -2202,8 +2256,10 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
            'title': 'lex will - Channels',
            'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
            'uploader': 'lex will',
            'uploader_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
        },
-        'playlist_mincount': 138,
+        'playlist_mincount': 12,
    }, {
        'url': 'https://invidio.us/channel/UCmlqkdCBesrv2Lak1mF_MxA',
        'only_matching': True,
@@ -2221,6 +2277,7 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            'id': 'PLwP_SiAcdui0KVebT0mU9Apz359a4ubsC',
            'uploader': 'Christiaan008',
            'uploader_id': 'UCEPzS1rYsrkqzSLNp76nrcg',
            'description': 'md5:a14dc1a8ef8307a9807fe136a0660268',
        },
        'playlist_count': 96,
    }, {
@@ -2255,6 +2312,7 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            'id': 'PLzH6n4zXuckpfMu_4Ff8E7Z1behQks5ba',
            'uploader_id': 'UC9-y-6csu5WGm29I7JiwpnA',
            'uploader': 'Computerphile',
            'description': 'md5:7f567c574d13d3f8c0954d9ffee4e487',
        },
        'playlist_mincount': 11,
    }, {
@@ -2295,12 +2353,12 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
        'info_dict': {
            'id': '9Auq9mYxFEE',
            'ext': 'mp4',
-            'title': 'Watch Sky News live',
+            'title': compat_str,
            'uploader': 'Sky News',
            'uploader_id': 'skynews',
            'uploader_url': r're:https?://(?:www\.)?youtube\.com/user/skynews',
            'upload_date': '20191102',
-            'description': 'md5:78de4e1c2359d0ea3ed829678e38b662',
+            'description': 'md5:85ddd75d888674631aaf9599a9a0b0ae',
            'categories': ['News & Politics'],
            'tags': list,
            'like_count': int,
@@ -2574,9 +2632,9 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
        next_continuation = cls._extract_next_continuation_data(renderer)
        if next_continuation:
            return next_continuation
-        contents = renderer.get('contents')
+        contents = []
-        if not isinstance(contents, list):
+        for key in ('contents', 'items'):
-            return
+            contents.extend(try_get(renderer, lambda x: x[key], list) or [])
        for content in contents:
            if not isinstance(content, dict):
                continue
@@ -2611,35 +2669,22 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                for isr_content in isr_contents:
                    if not isinstance(isr_content, dict):
                        continue
-                    renderer = isr_content.get('playlistVideoListRenderer')
+
-                    if renderer:
+                    known_renderers = {
-                        for entry in self._playlist_entries(renderer):
+                        'playlistVideoListRenderer': self._playlist_entries,
-                            yield entry
+                        'gridRenderer': self._grid_entries,
                        'shelfRenderer': lambda x: self._shelf_entries(x, tab.get('title') != 'Channels'),
                        'backstagePostThreadRenderer': self._post_thread_entries,
                        'videoRenderer': lambda x: [self._video_entry(x)],
                    }
                    for key, renderer in isr_content.items():
                        if key not in known_renderers:
                            continue
                        for entry in known_renderers[key](renderer):
                            if entry:
                                yield entry
                        continuation_list[0] = self._extract_continuation(renderer)
-                        continue
+                        break
                    renderer = isr_content.get('gridRenderer')
                    if renderer:
                        for entry in self._grid_entries(renderer):
                            yield entry
                        continuation_list[0] = self._extract_continuation(renderer)
                        continue
                    renderer = isr_content.get('shelfRenderer')
                    if renderer:
                        is_channels_tab = tab.get('title') == 'Channels'
                        for entry in self._shelf_entries(renderer, not is_channels_tab):
                            yield entry
                        continue
                    renderer = isr_content.get('backstagePostThreadRenderer')
                    if renderer:
                        for entry in self._post_thread_entries(renderer):
                            yield entry
                        continuation_list[0] = self._extract_continuation(renderer)
                        continue
                    renderer = isr_content.get('videoRenderer')
                    if renderer:
                        entry = self._video_entry(renderer)
                        if entry:
                            yield entry
                if not continuation_list[0]:
                    continuation_list[0] = self._extract_continuation(is_renderer)
@@ -2692,48 +2737,47 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
            if not response:
                break
            known_continuation_renderers = {
                'playlistVideoListContinuation': self._playlist_entries,
                'gridContinuation': self._grid_entries,
                'itemSectionContinuation': self._post_thread_continuation_entries,
                'sectionListContinuation': extract_entries,  # for feeds
            }
            continuation_contents = try_get(
-                response, lambda x: x['continuationContents'], dict)
+                response, lambda x: x['continuationContents'], dict) or {}
-            if continuation_contents:
+            continuation_renderer = None
-                continuation_renderer = continuation_contents.get('playlistVideoListContinuation')
+            for key, value in continuation_contents.items():
-                if continuation_renderer:
+                if key not in known_continuation_renderers:
                    for entry in self._playlist_entries(continuation_renderer):
                        yield entry
                    continuation = self._extract_continuation(continuation_renderer)
                    continue
                continuation_renderer = continuation_contents.get('gridContinuation')
                if continuation_renderer:
                    for entry in self._grid_entries(continuation_renderer):
                        yield entry
                    continuation = self._extract_continuation(continuation_renderer)
                    continue
                continuation_renderer = continuation_contents.get('itemSectionContinuation')
                if continuation_renderer:
                    for entry in self._post_thread_continuation_entries(continuation_renderer):
                        yield entry
                    continuation = self._extract_continuation(continuation_renderer)
                    continue
                continuation_renderer = continuation_contents.get('sectionListContinuation')  # for feeds
                if continuation_renderer:
                    continuation_list = [None]
                    for entry in extract_entries(continuation_renderer):
                        yield entry
                    continuation = continuation_list[0]
                    continue
                continuation_renderer = value
                continuation_list = [None]
                for entry in known_continuation_renderers[key](continuation_renderer):
                    yield entry
                continuation = continuation_list[0] or self._extract_continuation(continuation_renderer)
                break
            if continuation_renderer:
                continue
            known_renderers = {
                'gridPlaylistRenderer': (self._grid_entries, 'items'),
                'gridVideoRenderer': (self._grid_entries, 'items'),
                'playlistVideoRenderer': (self._playlist_entries, 'contents'),
                'itemSectionRenderer': (self._playlist_entries, 'contents'),
            }
            continuation_items = try_get(
                response, lambda x: x['onResponseReceivedActions'][0]['appendContinuationItemsAction']['continuationItems'], list)
-            if continuation_items:
+            continuation_item = try_get(continuation_items, lambda x: x[0], dict) or {}
-                continuation_item = continuation_items[0]
+            video_items_renderer = None
-                if not isinstance(continuation_item, dict):
+            for key, value in continuation_item.items():
-                    continue
+                if key not in known_renderers:
                renderer = continuation_item.get('playlistVideoRenderer') or continuation_item.get('itemSectionRenderer')
                if renderer:
                    video_list_renderer = {'contents': continuation_items}
                    for entry in self._playlist_entries(video_list_renderer):
                        yield entry
                    continuation = self._extract_continuation(video_list_renderer)
                    continue
                video_items_renderer = {known_renderers[key][1]: continuation_items}
                for entry in known_renderers[key][0](video_items_renderer):
                    yield entry
                continuation = self._extract_continuation(video_items_renderer)
                break
            if video_items_renderer:
                continue
            break
    @staticmethod
@@ -2784,12 +2828,15 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                data, lambda x: x['metadata']['playlistMetadataRenderer'], dict)
        if renderer:
            title = renderer.get('title')
-            description = renderer.get('description')
+            description = renderer.get('description', '')
            playlist_id = channel_id
            tags = renderer.get('keywords', '').split()
            thumbnails_list = (
                try_get(renderer, lambda x: x['avatar']['thumbnails'], list)
-                or data['sidebar']['playlistSidebarRenderer']['items'][0]['playlistSidebarPrimaryInfoRenderer']['thumbnailRenderer']['playlistVideoThumbnailRenderer']['thumbnail']['thumbnails']
+                or try_get(
                    data,
                    lambda x: x['sidebar']['playlistSidebarRenderer']['items'][0]['playlistSidebarPrimaryInfoRenderer']['thumbnailRenderer']['playlistVideoThumbnailRenderer']['thumbnail']['thumbnails'],
                    list)
                or [])
        thumbnails = []
@@ -3089,7 +3136,7 @@ class YoutubeFavouritesIE(YoutubeBaseInfoExtractor):
 class YoutubeSearchIE(SearchInfoExtractor, YoutubeBaseInfoExtractor):
-    IE_DESC = 'YouTube.com searches'
+    IE_DESC = 'YouTube.com searches, "ytsearch" keyword'
    # there doesn't appear to be a real limit, for example if you search for
    # 'python' you get more than 8.000.000 results
    _MAX_RESULTS = float('inf')
@@ -3134,6 +3181,12 @@ class YoutubeSearchIE(SearchInfoExtractor, YoutubeBaseInfoExtractor):
            # So we search through all entries till we find them.
            continuation_token = None
            for slr_content in slr_contents:
                if continuation_token is None:
                    continuation_token = try_get(
                        slr_content,
                        lambda x: x['continuationItemRenderer']['continuationEndpoint']['continuationCommand']['token'],
                        compat_str)
                isr_contents = try_get(
                    slr_content,
                    lambda x: x['itemSectionRenderer']['contents'],
@@ -3155,12 +3208,6 @@ class YoutubeSearchIE(SearchInfoExtractor, YoutubeBaseInfoExtractor):
                    if total == n:
                        return
                if continuation_token is None:
                    continuation_token = try_get(
                        slr_content,
                        lambda x: x['continuationItemRenderer']['continuationEndpoint']['continuationCommand']['token'],
                        compat_str)
            if not continuation_token:
                break
            data['continuation'] = continuation_token
@@ -3178,7 +3225,7 @@ class YoutubeSearchDateIE(YoutubeSearchIE):
 class YoutubeSearchURLIE(YoutubeSearchIE):
-    IE_DESC = 'YouTube.com searches, "ytsearch" keyword'
+    IE_DESC = 'YouTube.com search URLs'
    IE_NAME = YoutubeSearchIE.IE_NAME + '_url'
    _VALID_URL = r'https?://(?:www\.)?youtube\.com/results\?(.*?&)?(?:search_query|q)=(?:[^&]+)(?:[&]|$)'
    # _MAX_RESULTS = 100
@@ -18,14 +18,12 @@ from .utils import (
    get_executable_path,
    OUTTMPL_TYPES,
    preferredencoding,
    REMUX_EXTENSIONS,
    write_string,
 )
 from .version import __version__
 _remux_formats = ('mp4', 'mkv', 'flv', 'webm', 'mov', 'avi', 'mp3', 'mka', 'm4a', 'ogg', 'opus')
 def _hide_login_info(opts):
    PRIVATE_OPTS = set(['-p', '--password', '-u', '--username', '--video-password', '--ap-password', '--ap-username'])
    eqre = re.compile('^(?P<key>' + ('|'.join(re.escape(po) for po in PRIVATE_OPTS)) + ')=.+$')
@@ -155,7 +153,7 @@ def parseOpts(overrideArguments=None):
    general.add_option(
        '-U', '--update',
        action='store_true', dest='update_self',
-        help='[BROKEN] Update this program to latest version. Make sure that you have sufficient permissions (run with sudo if needed)')
+        help='Update this program to latest version. Make sure that you have sufficient permissions (run with sudo if needed)')
    general.add_option(
        '-i', '--ignore-errors', '--no-abort-on-error',
        action='store_true', dest='ignoreerrors', default=True,
@@ -521,6 +519,16 @@ def parseOpts(overrideArguments=None):
            'If a merge is required (e.g. bestvideo+bestaudio), '
            'output to given container format. One of mkv, mp4, ogg, webm, flv. '
            'Ignored if no merge is required'))
    video_format.add_option(
        '--allow-unplayable-formats',
        action='store_true', dest='allow_unplayable_formats', default=False,
        help=(
            'Allow unplayable formats to be listed and downloaded. '
            'All video postprocessing will also be turned off'))
    video_format.add_option(
        '--no-allow-unplayable-formats',
        action='store_false', dest='allow_unplayable_formats',
        help='Do not allow unplayable formats to be listed or downloaded (default)')
    subtitles = optparse.OptionGroup(parser, 'Subtitle Options')
    subtitles.add_option(
@@ -812,11 +820,13 @@ def parseOpts(overrideArguments=None):
    verbosity.add_option(
        '-C', '--call-home',
        dest='call_home', action='store_true', default=False,
-        help='[Broken] Contact the youtube-dlc server for debugging')
+        # help='[Broken] Contact the youtube-dlc server for debugging')
        help=optparse.SUPPRESS_HELP)
    verbosity.add_option(
        '--no-call-home',
        dest='call_home', action='store_false',
-        help='Do not contact the youtube-dlc server for debugging (default)')
+        # help='Do not contact the youtube-dlc server for debugging (default)')
        help=optparse.SUPPRESS_HELP)
    filesystem = optparse.OptionGroup(parser, 'Filesystem Options')
    filesystem.add_option(
@@ -1042,7 +1052,7 @@ def parseOpts(overrideArguments=None):
            'Remux the video into another container if necessary (currently supported: %s). '
            'If target container does not support the video/audio codec, remuxing will fail. '
            'You can specify multiple rules; eg. "aac>m4a/mov>mp4/mkv" will remux aac to m4a, mov to mp4 '
-            'and anything else to mkv.' % '|'.join(_remux_formats)))
+            'and anything else to mkv.' % '|'.join(REMUX_EXTENSIONS)))
    postproc.add_option(
        '--recode-video',
        metavar='FORMAT', dest='recodevideo', default=None,
@@ -10,9 +10,9 @@ import base64
 try:
    import mutagen
-    _has_mutagen = True
+    has_mutagen = True
 except ImportError:
-    _has_mutagen = False
+    has_mutagen = False
 from .ffmpeg import FFmpegPostProcessor
@@ -42,13 +42,12 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
    def run(self, info):
        filename = info['filepath']
        temp_filename = prepend_extension(filename, 'temp')
        files_to_delete = []
        if not info.get('thumbnails'):
            self.to_screen('There aren\'t any thumbnails to embed')
            return [], info
-        thumbnail_filename = info['thumbnails'][-1]['filename']
+        original_thumbnail = thumbnail_filename = info['thumbnails'][-1]['filename']
        if not os.path.exists(encodeFilename(thumbnail_filename)):
            self.report_warning('Skipping embedding the thumbnail because the file is missing.')
@@ -67,7 +66,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
                self.to_screen('Correcting extension to webp and escaping path for thumbnail "%s"' % thumbnail_filename)
                thumbnail_webp_filename = replace_extension(thumbnail_filename, 'webp')
                os.rename(encodeFilename(thumbnail_filename), encodeFilename(thumbnail_webp_filename))
-                thumbnail_filename = thumbnail_webp_filename
+                original_thumbnail = thumbnail_filename = thumbnail_webp_filename
                thumbnail_ext = 'webp'
        # Convert unsupported thumbnail formats to JPEG (see #25687, #25717)
@@ -79,9 +78,9 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
            escaped_thumbnail_jpg_filename = replace_extension(escaped_thumbnail_filename, 'jpg')
            self.to_screen('Converting thumbnail "%s" to JPEG' % escaped_thumbnail_filename)
            self.run_ffmpeg(escaped_thumbnail_filename, escaped_thumbnail_jpg_filename, ['-bsf:v', 'mjpeg2jpeg'])
            files_to_delete.append(escaped_thumbnail_filename)
            thumbnail_jpg_filename = replace_extension(thumbnail_filename, 'jpg')
            # Rename back to unescaped for further processing
            os.rename(encodeFilename(escaped_thumbnail_filename), encodeFilename(thumbnail_filename))
            os.rename(encodeFilename(escaped_thumbnail_jpg_filename), encodeFilename(thumbnail_jpg_filename))
            thumbnail_filename = thumbnail_jpg_filename
            thumbnail_ext = 'jpg'
@@ -153,7 +152,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
                    success = False
        elif info['ext'] in ['ogg', 'opus']:
-            if not _has_mutagen:
+            if not has_mutagen:
                raise EmbedThumbnailPPError('module mutagen was not found. Please install using `python -m pip install mutagen`')
            self.to_screen('Adding thumbnail to "%s"' % filename)
@@ -184,9 +183,11 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
        if success and temp_filename != filename:
            os.remove(encodeFilename(filename))
            os.rename(encodeFilename(temp_filename), encodeFilename(filename))
        files_to_delete = [thumbnail_filename]
        if self._already_have_thumbnail:
-            info['__files_to_move'][thumbnail_filename] = replace_extension(
+            info['__files_to_move'][original_thumbnail] = replace_extension(
-                info['__thumbnail_filename'], os.path.splitext(thumbnail_filename)[1][1:])
+                info['__thumbnail_filename'], os.path.splitext(original_thumbnail)[1][1:])
-        else:
+            if original_thumbnail == thumbnail_filename:
-            files_to_delete.append(thumbnail_filename)
+                files_to_delete = []
        return files_to_delete, info
@@ -221,8 +221,7 @@ class FFmpegPostProcessor(PostProcessor):
        cmd += opts
        cmd.append(encodeFilename(self._ffmpeg_filename_argument(path), True))
-        if self._downloader.params.get('verbose', False):
+        self.write_debug('ffprobe command line: %s' % shell_quote(cmd))
            self._downloader.to_screen('[debug] ffprobe command line: %s' % shell_quote(cmd))
        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
        stdout, stderr = p.communicate()
        return json.loads(stdout.decode('utf-8', 'replace'))
@@ -261,7 +260,7 @@ class FFmpegPostProcessor(PostProcessor):
        stdout, stderr = process_communicate_or_kill(p)
        if p.returncode != 0:
            stderr = stderr.decode('utf-8', 'replace').strip()
-            if self._downloader.params.get('verbose', False):
+            if self.get_param('verbose', False):
                self.report_error(stderr)
            raise FFmpegPostProcessorError(stderr.split('\n')[-1])
        self.try_utime(out_path, oldest_mtime, oldest_mtime)
@@ -281,6 +280,8 @@ class FFmpegPostProcessor(PostProcessor):
 class FFmpegExtractAudioPP(FFmpegPostProcessor):
    COMMON_AUDIO_EXTENSIONS = ('wav', 'flac', 'm4a', 'aiff', 'mp3', 'ogg', 'mka', 'opus', 'wma')
    def __init__(self, downloader=None, preferredcodec=None, preferredquality=None, nopostoverwrites=False):
        FFmpegPostProcessor.__init__(self, downloader)
        if preferredcodec is None:
@@ -302,6 +303,10 @@ class FFmpegExtractAudioPP(FFmpegPostProcessor):
    def run(self, information):
        path = information['filepath']
        orig_ext = information['ext']
        if self._preferredcodec == 'best' and orig_ext in self.COMMON_AUDIO_EXTENSIONS:
            self.to_screen('Skipping audio extraction since the file is already in a common audio format')
        filecodec = self.get_audio_codec(path)
        if filecodec is None:
@@ -442,6 +447,10 @@ class FFmpegVideoConvertorPP(FFmpegPostProcessor):
 class FFmpegEmbedSubtitlePP(FFmpegPostProcessor):
    def __init__(self, downloader=None, already_have_subtitle=False):
        super(FFmpegEmbedSubtitlePP, self).__init__(downloader)
        self._already_have_subtitle = already_have_subtitle
    def run(self, information):
        if information['ext'] not in ('mp4', 'webm', 'mkv'):
            self.to_screen('Subtitles can only be embedded in mp4, webm or mkv files')
@@ -501,7 +510,8 @@ class FFmpegEmbedSubtitlePP(FFmpegPostProcessor):
        os.remove(encodeFilename(filename))
        os.rename(encodeFilename(temp_filename), encodeFilename(filename))
-        return sub_filenames, information
+        files_to_delete = [] if self._already_have_subtitle else sub_filenames
        return files_to_delete, information
 class FFmpegMetadataPP(FFmpegPostProcessor):
@@ -4,11 +4,11 @@ import shutil
 from .common import PostProcessor
 from ..utils import (
    decodeFilename,
    encodeFilename,
    make_dir,
    PostProcessingError,
 )
 from ..compat import compat_str
 class MoveFilesAfterDownloadPP(PostProcessor):
@@ -26,12 +26,12 @@ class MoveFilesAfterDownloadPP(PostProcessor):
        finaldir = info.get('__finaldir', dl_path)
        finalpath = os.path.join(finaldir, dl_name)
        self.files_to_move.update(info['__files_to_move'])
-        self.files_to_move[info['filepath']] = finalpath
+        self.files_to_move[info['filepath']] = decodeFilename(finalpath)
        make_newfilename = lambda old: decodeFilename(os.path.join(finaldir, os.path.basename(encodeFilename(old))))
        for oldfile, newfile in self.files_to_move.items():
            if not newfile:
-                newfile = os.path.join(finaldir, os.path.basename(encodeFilename(oldfile)))
+                newfile = make_newfilename(oldfile)
            oldfile, newfile = compat_str(oldfile), compat_str(newfile)
            if os.path.abspath(encodeFilename(oldfile)) == os.path.abspath(encodeFilename(newfile)):
                continue
            if not os.path.exists(encodeFilename(oldfile)):
@@ -50,5 +50,5 @@ class MoveFilesAfterDownloadPP(PostProcessor):
            self.to_screen('Moving file "%s" to "%s"' % (oldfile, newfile))
            shutil.move(oldfile, newfile)  # os.rename cannot move between volumes
-        info['filepath'] = compat_str(finalpath)
+        info['filepath'] = finalpath
        return [], info
@@ -12,6 +12,7 @@ from ..utils import (
    str_or_none,
    PostProcessingError,
    prepend_extension,
    process_communicate_or_kill,
 )
@@ -43,6 +44,10 @@ class SponSkrubPP(PostProcessor):
        if self.path is None:
            return [], information
        filename = information['filepath']
        if not os.path.exists(encodeFilename(filename)):  # no download
            return [], information
        if information['extractor_key'].lower() != 'youtube':
            self.to_screen('Skipping sponskrub since it is not a YouTube video')
            return [], information
@@ -58,7 +63,6 @@ class SponSkrubPP(PostProcessor):
            if not information.get('__real_download', False):
                self.report_warning('If sponskrub is run multiple times, unintended parts of the video could be cut out.')
        filename = information['filepath']
        temp_filename = prepend_extension(filename, self._temp_ext)
        if os.path.exists(encodeFilename(temp_filename)):
            os.remove(encodeFilename(temp_filename))
@@ -72,8 +76,9 @@ class SponSkrubPP(PostProcessor):
        cmd = [encodeArgument(i) for i in cmd]
        self.write_debug('sponskrub command line: %s' % shell_quote(cmd))
-        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
+        pipe = None if self.get_param('verbose') else subprocess.PIPE
-        stdout, stderr = p.communicate()
+        p = subprocess.Popen(cmd, stdout=pipe)
        stdout = process_communicate_or_kill(p)[0]
        if p.returncode == 0:
            os.remove(encodeFilename(filename))
@@ -82,9 +87,7 @@ class SponSkrubPP(PostProcessor):
        elif p.returncode == 3:
            self.to_screen('No segments in the SponsorBlock database')
        else:
-            msg = stderr.decode('utf-8', 'replace').strip() or stdout.decode('utf-8', 'replace').strip()
+            msg = stdout.decode('utf-8', 'replace').strip() if stdout else ''
-            self.write_debug(msg, prefix=False)
+            msg = msg.split('\n')[0 if msg.lower().startswith('unrecognised') else -1]
            line = 0 if msg[:12].lower() == 'unrecognised' else -1
            msg = msg.split('\n')[line]
            raise PostProcessingError(msg if msg else 'sponskrub failed with error code %s' % p.returncode)
        return [], information
@@ -5,6 +5,7 @@ import json
 import traceback
 import hashlib
 import os
 import platform
 import subprocess
 import sys
 from zipimport import zipimporter
@@ -15,6 +16,7 @@ from .utils import encode_compat_str
 from .version import __version__
 '''  # Not signed
 def rsa_verify(message, signature, key):
    from hashlib import sha256
    assert isinstance(message, bytes)
@@ -27,82 +29,79 @@ def rsa_verify(message, signature, key):
        return False
    expected = b'0001' + (byte_size - len(asn1) // 2 - 3) * b'ff' + b'00' + asn1
    return expected == signature
 '''
 def update_self(to_screen, verbose, opener):
-    """Update the program file with the latest version from the repository"""
+    """
    Update the program file with the latest version from the repository
    Returns whether the program should terminate
    """
-    return to_screen('Update is currently broken.\nVisit  https://github.com/pukkandan/yt-dlp/releases/latest  to get the latest version')
+    JSON_URL = 'https://api.github.com/repos/pukkandan/yt-dlp/releases/latest'
-    UPDATE_URL = 'https://blackjack4494.github.io//update/'
+    def calc_sha256sum(path):
    VERSION_URL = UPDATE_URL + 'LATEST_VERSION'
    JSON_URL = UPDATE_URL + 'versions.json'
    UPDATES_RSA_KEY = (0x9d60ee4d8f805312fdb15a62f87b95bd66177b91df176765d13514a0f1754bcd2057295c5b6f1d35daa6742c3ffc9a82d3e118861c207995a8031e151d863c9927e304576bc80692bc8e094896fcf11b66f3e29e04e3a71e9a11558558acea1840aec37fc396fb6b65dc81a1c4144e03bd1c011de62e3f1357b327d08426fe93, 65537)
    def sha256sum():
        h = hashlib.sha256()
        b = bytearray(128 * 1024)
        mv = memoryview(b)
-        with open(os.path.realpath(sys.executable), 'rb', buffering=0) as f:
+        with open(os.path.realpath(path), 'rb', buffering=0) as f:
            for n in iter(lambda: f.readinto(mv), 0):
                h.update(mv[:n])
        return h.hexdigest()
-    to_screen('Current Build Hash %s' % sha256sum())
+    to_screen('Current Build Hash %s' % calc_sha256sum(sys.executable))
    if not isinstance(globals().get('__loader__'), zipimporter) and not hasattr(sys, 'frozen'):
-        to_screen('It looks like you installed youtube-dlc with a package manager, pip, setup.py or a tarball. Please use that to update.')
+        to_screen('It looks like you installed yt-dlp with a package manager, pip, setup.py or a tarball. Please use that to update.')
        return
    # compiled file.exe can find itself by
    # to_screen(os.path.basename(sys.executable))
    # and path to py or exe
    # to_screen(os.path.realpath(sys.executable))
    # Check if there is a new version
    try:
        newversion = opener.open(VERSION_URL).read().decode('utf-8').strip()
    except Exception:
        if verbose:
            to_screen(encode_compat_str(traceback.format_exc()))
        to_screen('ERROR: can\'t find the current version. Please try again later.')
        to_screen('Visit https://github.com/blackjack4494/yt-dlc/releases/latest')
        return
    if newversion == __version__:
        to_screen('youtube-dlc is up-to-date (' + __version__ + ')')
        return
    # Download and check versions info
    try:
-        versions_info = opener.open(JSON_URL).read().decode('utf-8')
+        version_info = opener.open(JSON_URL).read().decode('utf-8')
-        versions_info = json.loads(versions_info)
+        version_info = json.loads(version_info)
    except Exception:
        if verbose:
            to_screen(encode_compat_str(traceback.format_exc()))
        to_screen('ERROR: can\'t obtain versions info. Please try again later.')
-        to_screen('Visit https://github.com/blackjack4494/yt-dlc/releases/latest')
+        to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
        return
    if 'signature' not in versions_info:
        to_screen('ERROR: the versions file is not signed or corrupted. Aborting.')
        return
    signature = versions_info['signature']
    del versions_info['signature']
    if not rsa_verify(json.dumps(versions_info, sort_keys=True).encode('utf-8'), signature, UPDATES_RSA_KEY):
        to_screen('ERROR: the versions file signature is invalid. Aborting.')
        return
    version_id = versions_info['latest']
    def version_tuple(version_str):
        return tuple(map(int, version_str.split('.')))
    version_id = version_info['tag_name']
    if version_tuple(__version__) >= version_tuple(version_id):
-        to_screen('youtube-dlc is up to date (%s)' % __version__)
+        to_screen('yt-dlp is up to date (%s)' % __version__)
        return
    to_screen('Updating to version ' + version_id + ' ...')
    version = versions_info['versions'][version_id]
-    print_notes(to_screen, versions_info['versions'])
+    version_labels = {
        'zip_3': '',
        'zip_2': '',
        # 'zip_2': '_py2',
        'exe_64': '.exe',
        'exe_32': '_x86.exe',
    }
    def get_bin_info(bin_or_exe, version):
        label = version_labels['%s_%s' % (bin_or_exe, version)]
        return next(
            (i for i in version_info['assets']
                if i['name'] in ('yt-dlp%s' % label, 'youtube-dlc%s' % label)), {})
    def get_sha256sum(bin_or_exe, version):
        label = version_labels['%s_%s' % (bin_or_exe, version)]
        urlh = next(
            (i for i in version_info['assets']
                if i['name'] in ('SHA2-256SUMS')), {}).get('browser_download_url')
        if not urlh:
            return None
        hash_data = opener.open(urlh).read().decode('utf-8')
        hashes = list(map(lambda x: x.split(':'), hash_data.splitlines()))
        return next(
            (i[1] for i in hashes
                if i[0] in ('yt-dlp%s' % label, 'youtube-dlc%s' % label)), None)
    # sys.executable is set to the full pathname of the exe-file for py2exe
    # though symlinks are not followed so that we need to do this manually
@@ -113,7 +112,7 @@ def update_self(to_screen, verbose, opener):
        to_screen('ERROR: no write permissions on %s' % filename)
        return
-    # Py2EXE
+    # PyInstaller
    if hasattr(sys, 'frozen'):
        exe = filename
        directory = os.path.dirname(exe)
@@ -122,19 +121,20 @@ def update_self(to_screen, verbose, opener):
            return
        try:
-            urlh = opener.open(version['exe'][0])
+            arch = platform.architecture()[0][:2]
            url = get_bin_info('exe', arch).get('browser_download_url')
            if not url:
                to_screen('ERROR: unable to fetch updates')
                to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
                return
            urlh = opener.open(url)
            newcontent = urlh.read()
            urlh.close()
-        except (IOError, OSError):
+        except (IOError, OSError, StopIteration):
            if verbose:
                to_screen(encode_compat_str(traceback.format_exc()))
            to_screen('ERROR: unable to download latest version')
-            to_screen('Visit https://github.com/blackjack4494/yt-dlc/releases/latest')
+            to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
            return
        newcontent_hash = hashlib.sha256(newcontent).hexdigest()
        if newcontent_hash != version['exe'][1]:
            to_screen('ERROR: the downloaded file hash does not match. Aborting.')
            return
        try:
@@ -146,20 +146,33 @@ def update_self(to_screen, verbose, opener):
            to_screen('ERROR: unable to write the new version')
            return
        expected_sum = get_sha256sum('exe', arch)
        if not expected_sum:
            to_screen('WARNING: no hash information found for the release')
        elif calc_sha256sum(exe + '.new') != expected_sum:
            to_screen('ERROR: unable to verify the new executable')
            to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
            try:
                os.remove(exe + '.new')
            except OSError:
                to_screen('ERROR: unable to remove corrupt download')
            return
        try:
-            bat = os.path.join(directory, 'youtube-dlc-updater.bat')
+            bat = os.path.join(directory, 'yt-dlp-updater.cmd')
            with io.open(bat, 'w') as batfile:
                batfile.write('''
-@echo off
+@(
-echo Waiting for file handle to be closed ...
+    echo.Waiting for file handle to be closed ...
-ping 127.0.0.1 -n 5 -w 1000 > NUL
+    ping 127.0.0.1 -n 5 -w 1000 > NUL
-move /Y "%s.new" "%s" > NUL
+    move /Y "%s.new" "%s" > NUL
-echo Updated youtube-dlc to version %s.
+    echo.Updated yt-dlp to version %s.
-start /b "" cmd /c del "%%~f0"&exit /b"
+)
-                \n''' % (exe, exe, version_id))
+@start /b "" cmd /c del "%%~f0"&exit /b
                ''' % (exe, exe, version_id))
            subprocess.Popen([bat])  # Continues to run in the background
-            return  # Do not show premature success messages
+            return True  # Exit app
        except (IOError, OSError):
            if verbose:
                to_screen(encode_compat_str(traceback.format_exc()))
@@ -169,33 +182,51 @@ start /b "" cmd /c del "%%~f0"&exit /b"
    # Zip unix package
    elif isinstance(globals().get('__loader__'), zipimporter):
        try:
-            urlh = opener.open(version['bin'][0])
+            py_ver = platform.python_version()[0]
            url = get_bin_info('zip', py_ver).get('browser_download_url')
            if not url:
                to_screen('ERROR: unable to fetch updates')
                to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
                return
            urlh = opener.open(url)
            newcontent = urlh.read()
            urlh.close()
-        except (IOError, OSError):
+        except (IOError, OSError, StopIteration):
            if verbose:
                to_screen(encode_compat_str(traceback.format_exc()))
            to_screen('ERROR: unable to download latest version')
-            to_screen('Visit https://github.com/blackjack4494/yt-dlc/releases/latest')
+            to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
            return
        newcontent_hash = hashlib.sha256(newcontent).hexdigest()
        if newcontent_hash != version['bin'][1]:
            to_screen('ERROR: the downloaded file hash does not match. Aborting.')
            return
        try:
-            with open(filename, 'wb') as outf:
+            with open(filename + '.new', 'wb') as outf:
                outf.write(newcontent)
        except (IOError, OSError):
            if verbose:
                to_screen(encode_compat_str(traceback.format_exc()))
            to_screen('ERROR: unable to write the new version')
            return
        expected_sum = get_sha256sum('zip', py_ver)
        if expected_sum and calc_sha256sum(filename + '.new') != expected_sum:
            to_screen('ERROR: unable to verify the new zip')
            to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/latest')
            try:
                os.remove(filename + '.new')
            except OSError:
                to_screen('ERROR: unable to remove corrupt zip')
            return
        try:
            os.rename(filename + '.new', filename)
        except OSError:
            to_screen('ERROR: unable to overwrite current version')
            return
-    to_screen('Updated youtube-dlc. Restart youtube-dlc to use the new version.')
+    to_screen('Updated yt-dlp. Restart youtube-dlc to use the new version.')
 '''  # UNUSED
 def get_notes(versions, fromVersion):
    notes = []
    for v, vdata in sorted(versions.items()):
@@ -210,3 +241,4 @@ def print_notes(to_screen, versions, fromVersion=__version__):
        to_screen('PLEASE NOTE:')
        for note in notes:
            to_screen(note)
 '''
@@ -1715,6 +1715,8 @@ KNOWN_EXTENSIONS = (
    'wav',
    'f4f', 'f4m', 'm3u8', 'smil')
 REMUX_EXTENSIONS = ('mp4', 'mkv', 'flv', 'webm', 'mov', 'avi', 'mp3', 'mka', 'm4a', 'ogg', 'opus')
 # needed for sanitizing filenames in restricted mode
 ACCENT_CHARS = dict(zip('ÂÃÄÀÁÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖŐØŒÙÚÛÜŰÝÞßàáâãäåæçèéêëìíîïðñòóôõöőøœùúûüűýþÿ',
                        itertools.chain('AAAAAA', ['AE'], 'CEEEEIIIIDNOOOOOOO', ['OE'], 'UUUUUY', ['TH', 'ss'],
@@ -2607,6 +2609,8 @@ class YoutubeDLHandler(compat_urllib_request.HTTPHandler):
    @staticmethod
    def deflate(data):
        if not data:
            return data
        try:
            return zlib.decompress(data, -zlib.MAX_WBITS)
        except zlib.error:
@@ -4690,9 +4694,7 @@ def cli_configuration_args(params, arg_name, key, default=[], exe=None):  # retu
        return default, False
    assert isinstance(argdict, dict)
    assert isinstance(key, compat_str)
    key = key.lower()
    args = exe_args = None
    if exe is not None:
        assert isinstance(exe, compat_str)
@@ -5934,7 +5936,7 @@ def make_dir(path, to_screen=None):
 def get_executable_path():
    path = os.path.dirname(sys.argv[0])
-    if os.path.abspath(sys.argv[0]) != os.path.abspath(sys.executable):  # Not packaged
+    if os.path.basename(sys.argv[0]) == '__main__':  # Running from source
        path = os.path.join(path, '..')
    return os.path.abspath(path)
@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
-__version__ = '2021.01.29'
+__version__ = '2021.02.09'
Author	SHA1	Message	Date
pukkandan	aa837ddf06	Release 2021.02.15	2021-02-16 04:04:27 +05:30
pukkandan	a718ef84c8	[youtube] Fix for new accounts Cookies for some new accounts doesn't work with age-gated videos without `has_verified=1`	2021-02-16 03:20:06 +05:30
shirt-dev	44f705d001	#88 Implement SHA256 checking for autoupdater * Also fix bugs from `e5813e53f0` Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com> :ci skip dl	2021-02-16 02:36:42 +05:30
shirt-dev	47930b73a5	Fix build.yml hashing and crypto support (#87 ) Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com>	2021-02-16 00:46:23 +05:30
pukkandan	1de75fa129	[ExtractAudio] Don't re-encode when file is already in a common audio format (Closes #58 ) Fixes: https://github.com/blackjack4494/youtube-dlc/issues/214 Fixes: https://github.com/ytdl-org/youtube-dl/issues/28006	2021-02-15 23:22:11 +05:30
pukkandan	6285297795	[rumble] Add support for video page (Closes #80 )	2021-02-15 20:08:27 +05:30
pukkandan	e5813e53f0	Improve build/updater * Fix `get_executable_path` in UNIX * Update `x86.exe` correctly * Exit immediately in windows once the update process starts so that the file handle is released correctly * Show `exe`/`zip`/`source` and 32/64bit in verbose message * Look for both `yt-dlp` and `youtube-dlc` in releases. This ensures that the updater will keep working when the binary name is changed to yt-dlp * Disable pycryptodome in win_x86 since it causes `distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 10.0 is required`	2021-02-15 15:41:40 +05:30
siikamiika	273762c8d0	#86 [youtube_live_chat] Use POST API (Closes #82 ) YouTube has removed support for the old GET based live chat API, and it's now returning 404 Authored by siikamiika	2021-02-15 15:27:21 +05:30
shirt-dev	7620cd46c3	#79 Fix HLS AES-128 with multiple keys in external downloaders Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com>	2021-02-13 21:45:41 +05:30
pukkandan	068693675e	Cleanup some code and fix typos :ci skip dl	2021-02-12 20:32:49 +05:30
pukkandan	1ea2412927	Minor bugfixes * `__real_download` should be false when ffmpeg unavailable and no download * Mistakes in #70 * `allow_playlist_files` was not correctly pass through	2021-02-12 20:29:29 +05:30
shirt-dev	63ad4d43eb	#70 Allow downloading of unplayable video formats Video postprocessors are also turned off when this option is used Co-authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com> Co-authored-by: pukkandan <pukkandan@gmail.com>	2021-02-12 09:21:59 +05:30
pukkandan	584bab3766	[sponskrub] Print ffmpeg output and errors to terminal The ffmpeg run can be long when using `--sponskrub-cut`. So progress needs to be printed :ci skip dl	2021-02-12 01:40:08 +05:30
shirt-dev	fc2119f210	#76 Fix for empty HTTP head requests Related: https://github.com/ytdl-org/youtube-dl/issues/7181 Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com> (shirt-dev)	2021-02-11 21:31:34 +05:30
shirt-dev	5d25607a3a	#75 Change optional dependency from `Crypto` to `pycryptodome` (Closes #74 ) Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com> (shirt-dev) pycryptodome is an in-place replacement for Crypto and is more actively developed	2021-02-11 17:16:02 +05:30
pukkandan	a96c6d154a	[youtube] Fix search continuations	2021-02-11 17:10:38 +05:30
pukkandan	cc2db87805	Update to ytdl-2021.02.10 Except: [archiveorg] Fix and improve extraction (5fc53690cbe6abb11941a3f4846b566a7472753e)	2021-02-11 03:03:39 +05:30
shirt-dev	539d158c50	#72 Fix issue with unicode filenames in aria2c (Closes #71 ) Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com> (shirt-dev)	2021-02-11 02:27:18 +05:30
kurumigi	fb198a8a9c	#49 [niconico] Improved extraction and support encrypted/SMILE movies Co-authored-by: tsukumijima <tsukumijima@users.noreply.github.com> Co-authored-by: tsukumi <39271166+tsukumijima@users.noreply.github.com> Co-authored-by: Bepis <36346617+bbepis@users.noreply.github.com> Co-authored-by: pukkandan <pukkandan@gmail.com>	2021-02-10 12:15:20 +05:30
pukkandan	8d801631cf	[version] update :ci skip all	2021-02-10 02:06:49 +05:30
pukkandan	ba9f36d732	Release 2021.02.09	2021-02-10 01:26:56 +05:30
pukkandan	cffab0eefc	[embedsubtitle] Keep original subtitle after conversion if write_subtitles given Closes: https://github.com/pukkandan/yt-dlp/issues/57#issuecomment-775227745 :ci skip dl	2021-02-10 00:12:42 +05:30
pukkandan	2e339f59c3	[embedthumbnail] Keep original thumbnail after conversion if write_thumbnail given (Closes #67 ) Closes https://github.com/ytdl-org/youtube-dl/issues/27041 :ci skip dl	2021-02-09 23:18:20 +05:30
pukkandan	6c4fd172de	Add fallback for thumbnails Workaround for: https://github.com/ytdl-org/youtube-dl/issues/28023 Related: https://github.com/ytdl-org/youtube-dl/pull/28031 Also fixes https://www.reddit.com/r/youtubedl/comments/lfslw1/youtubedlp_with_aria2c_for_dash_support_is/gmolt0r?context=3	2021-02-09 23:12:41 +05:30
pukkandan	deaec5afc2	[youtube] Fix tests	2021-02-09 22:01:34 +05:30
pukkandan	69184e4152	[youtube] Simplified renderer parsing	2021-02-09 21:37:59 +05:30
pukkandan	a1b535bd75	[youtube] Support gridPlaylistRenderer and gridVideoRenderer (Closes #65 )	2021-02-09 20:40:37 +05:30
pukkandan	b3943b2f33	[pyinst.py] Move back to root dir (Closes #63 )	2021-02-09 18:04:27 +05:30
shirt-dev	3dd264bf42	#64 Implement self updater Co-authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com> (shirt-dev) Co-authored-by: pukkandan <pukkandan@gmail.com>	2021-02-09 18:04:00 +05:30
pukkandan	efabc16165	[postprocessor] Fix bug (Closes #62 ) introduced by: `1bf540d28b` :ci skip dl	2021-02-09 00:27:39 +05:30
shirt-dev	5219cb3e75	#55 Add aria2c support for DASH (mpd) and HLS (m3u8) Co-authored-by: Dan <2660574+shirtjs@users.noreply.github.com> Co-authored-by: pukkandan <pukkandan@gmail.com>	2021-02-08 22:16:01 +05:30
pukkandan	ff84930c86	[youtube] Bugfix (Closes #60 )	2021-02-08 19:20:19 +05:30
pukkandan	06ff212d64	[documentation] Crypto is an optional dependency	2021-02-08 18:05:22 +05:30
pukkandan	1bf540d28b	[sponskrub] Don't raise error when the video does not exist Eg: `--convert-sub srt --no-download --sponskrub` gave error before :ci skip dl	2021-02-08 15:48:12 +05:30
pukkandan	df692c5a7a	[remuxvideo] Fix validation of conditional remux	2021-02-08 15:29:02 +05:30
pukkandan	ecc97af344	[youtube] Don't show warning for empty playlist description (Closes #54 ) :ci skip dl	2021-02-07 20:15:02 +05:30
pukkandan	8a0b932258	[movefiles] Fix compatibility with python2 :ci skip dl	2021-02-07 17:41:41 +05:30
pukkandan	4d608b522f	[youtube_live_chat] Improve extraction :ci skip dl	2021-02-07 15:22:36 +05:30
pukkandan	885d36d4e4	[youtube] Fix comment extraction (Closes #53 ) :ci skip dl	2021-02-05 16:47:44 +05:30
pukkandan	0fd1a2b0bf	[version] update (and linter)	2021-02-05 05:02:41 +05:30
`@@ -1,3 +1,3 @@`
	`from __future__ import unicode_literals`	`from __future__ import unicode_literals`

	`__version__ = '2021.01.29'`	`__version__ = '2021.02.09'`