Update docs.html

This commit also removed the old empty chat filtering logic.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -8,9 +8,13 @@ assignees: ''
 ---
 
 # Must have
-- WhatsApp version: [WhatsApp version] - [Android/iOS]
+- WhatsApp version: [WhatsApp version]
+- OS: [Android/iOS] - [version]
 - Platform: [Linux/Windows/MacOS]
-- Branch and version: [main/dev] - [exporter version]
+- Exporter's branch and version: [main/dev] - [exporter version]
+
+**Describe the bug**
+A clear and concise description of what the bug is.
 
 If it is an error yield by Python, please also provide the trackback
 ```
@@ -18,8 +22,6 @@ If it is an error yield by Python, please also provide the trackback
 ```
 
 # Nice to have
-**Describe the bug**
-A clear and concise description of what the bug is.
 
 **To Reproduce**
 Steps to reproduce the behavior:

.github/workflows/compile-binary.yml

@@ -12,22 +12,22 @@ jobs:
   linux:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
     - name: Set up Python
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v5
       with:
-        python-version: '3.11'
+        python-version: '3.12'
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install pycryptodome javaobj-py3 ordered-set zstandard nuitka
+        pip install pycryptodome javaobj-py3 ordered-set zstandard nuitka==2.3
         pip install .
     - name: Build binary with Nuitka
       run: |
         python -m nuitka --no-deployment-flag=self-execution --onefile --include-data-file=./Whatsapp_Chat_Exporter/whatsapp.html=./Whatsapp_Chat_Exporter/whatsapp.html --follow-imports Whatsapp_Chat_Exporter/__main__.py 
         cp __main__.bin wtsexporter_linux_x64
         sha256sum wtsexporter_linux_x64
-    - uses: actions/upload-artifact@v3
+    - uses: actions/upload-artifact@v4
       with:
         name: binary-linux
         path: |
@@ -36,22 +36,22 @@ jobs:
   windows:
     runs-on: windows-latest
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
     - name: Set up Python
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v5
       with:
-        python-version: '3.11'
+        python-version: '3.12'
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install pycryptodome javaobj-py3 ordered-set zstandard nuitka
+        pip install pycryptodome javaobj-py3 ordered-set zstandard nuitka==2.3
         pip install .
     - name: Build binary with Nuitka
       run: |
         python -m nuitka --no-deployment-flag=self-execution --onefile --include-data-file=./Whatsapp_Chat_Exporter/whatsapp.html=./Whatsapp_Chat_Exporter/whatsapp.html --assume-yes-for-downloads --follow-imports Whatsapp_Chat_Exporter\__main__.py 
         copy __main__.exe wtsexporter_x64.exe
         Get-FileHash wtsexporter_x64.exe
-    - uses: actions/upload-artifact@v3
+    - uses: actions/upload-artifact@v4
       with:
         name: binary-windows
         path: |
@@ -60,22 +60,22 @@ jobs:
   macos:
     runs-on: macos-latest
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
     - name: Set up Python
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v5
       with:
-        python-version: '3.10'
+        python-version: '3.12'
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install pycryptodome javaobj-py3 ordered-set zstandard nuitka
+        pip install pycryptodome javaobj-py3 ordered-set zstandard nuitka==2.3
         pip install .
     - name: Build binary with Nuitka
       run: |
         python -m nuitka --no-deployment-flag=self-execution --onefile --include-data-file=./Whatsapp_Chat_Exporter/whatsapp.html=./Whatsapp_Chat_Exporter/whatsapp.html --follow-imports Whatsapp_Chat_Exporter/__main__.py 
         cp __main__.bin wtsexporter_macos_x64
         shasum -a 256 wtsexporter_macos_x64
-    - uses: actions/upload-artifact@v3
+    - uses: actions/upload-artifact@v4
       with:
         name: binary-macos
         path: |

.github/workflows/python-publish.yml

@@ -11,6 +11,10 @@ name: Upload Python Package
 on:
   release:
     types: [published]
+  workflow_dispatch:
+
+permissions:
+  id-token: write
 
 jobs:
   deploy:

.gitignore

@@ -134,3 +134,14 @@ dmypy.json
 *.onefile-build/
 *.exe
 __main__
+
+
+# Dev time intermidiates & temp files
+result/
+WhatsApp/
+/*.db
+/*.db-*
+/myout
+/msgstore.db
+/myout-json
+.vscode/

README.md

@@ -4,12 +4,15 @@
 [![Python](https://img.shields.io/pypi/pyversions/Whatsapp-Chat-Exporter)](https://pypi.org/project/Whatsapp-Chat-Exporter/)
 
 A customizable Android and iPhone Whatsapp database parser that will give you the history of your Whatsapp conversations in HTML and JSON. Inspired by [Telegram Chat Export Tool](https://telegram.org/blog/export-and-more).  
-**If you plan to uninstall WhatsApp or delete your WhatsApp account, please make a backup of your WhatsApp database. You may want to use this exporter again on the same database in the future as the exporter develops**
+> [!TIP]
+> If you plan to uninstall WhatsApp or delete your WhatsApp account, please make a backup of your WhatsApp database. You may want to use this exporter again on the same database in the future as the exporter develops.
+
+If you would like to support this project, all you need to do is to contribute or share this project! If you think otherwise and want to make a donation, please refer to the [Donation Guide](https://blog.knugi.com/DONATE.html).
 
 # Usage
-**Usage in README may be removed in the future. Check the usage in [Wiki](https://github.com/KnugiHK/Whatsapp-Chat-Exporter/wiki)**.
 
-**If you want to use the old release (< 0.5) of the exporter, please follow the [old usage guide](https://github.com/KnugiHK/Whatsapp-Chat-Exporter/wiki/Old-Usage#usage)**.
+> [!NOTE]
+> Usage in README may be removed in the future. Check the usage in [Wiki](https://github.com/KnugiHK/Whatsapp-Chat-Exporter/wiki)
 
 First, install the exporter by:
 ```shell
@@ -21,6 +24,10 @@ Then, create a working directory in somewhere you want
 mkdir working_wts
 cd working_wts
 ```
+
+> [!TIP]
+> macOS users should grant *Full Disk Access* to Terminal in the *Security & Privacy* settings before using the exporter.
+
 ## Working with Android
 ### Unencrypted WhatsApp database
 Extract the WhatsApp database with whatever means, one possible means is to use the [WhatsApp-Key-DB-Extractor](https://github.com/KnugiHK/WhatsApp-Key-DB-Extractor)
@@ -35,6 +42,13 @@ Simply invoke the following command from shell.
 ```sh
 wtsexporter -a
 ```
+#### Enriching Contact from vCard
+Usually, the default WhatsApp contact database extracted from your phone will contains the contact names and the exporter will use it to map your chats. However, some reported cases showed that the database could has never been populated. 
+In this case, you can export your contacts to a vCard file from your phone or a cloud provider like Google Contacts. Then, install the necessary dependency and run the following command from the shell:
+```sh
+pip install whatsapp-chat-exporter["vcards"]
+wtsexporter -a --enrich-from-vcard contacts.vcf --default-country-code 852
+```
 
 ### Encrypted Android WhatsApp Backup
 In order to support the decryption, install pycryptodome if it is not installed
@@ -42,7 +56,10 @@ In order to support the decryption, install pycryptodome if it is not installed
 pip install pycryptodome # Or 
 pip install whatsapp-chat-exporter["android_backup"] # install along with this software
 ```
-### Crypt15 is now the easiest way to decrypt a backup. If you have the 32 bytes hex key generated when you enable End-to-End encrypted backup, you can use it to decrypt the backup. If you do not have the 32 bytes hex key, you can still use the key file extracted just like extacting key file for Crypt12 and Crypt14 to decrypt the backup.
+
+> [!TIP]
+> Crypt15 is now the easiest way to decrypt a backup. If you have the 32 bytes hex key generated when you enable End-to-End encrypted backup, you can use it to decrypt the backup. If you do not have the 32 bytes hex key, you can still use the key file extracted just like extacting key file for Crypt12 and Crypt14 to decrypt the backup.
+
 #### Crypt12 or Crypt14
 You will need the decryption key file from your phone. If you have root access, you can find it as `/data/data/com.whatsapp/files/key`. Otherwise, if you used WhatsApp-Key-DB-Extractor before, it will appear in the WhatsApp backup directory as `WhatsApp/Databases/.nomedia`.
 
@@ -63,7 +80,10 @@ To support Crypt15 backup, install javaobj-py3 if it is not installed
 pip install javaobj-py3 # Or 
 pip install whatsapp-chat-exporter["crypt15"] # install along with this software
 ```
-Place the encrypted WhatsApp Backup (msgstore.db.crypt15) in the working directory. If you also want the name of your contacts, get the contact database, which is called wa.db. And copy the WhatsApp (Media) directory from your phone directly.  
+Before proceeding with this method, you must first create an end-to-end encrypted backup. For detailed instructions, refer to [WhatsApp's help center](https://faq.whatsapp.com/490592613091019).
+
+Once you have copied the backup files to your computer, place the encrypted WhatsApp backup file (msgstore.db.crypt15) into the working directory. If you also wish to include your contacts' names, obtain the contact database file, named wa.db. Additionally, copy the WhatsApp Media folder directly from your phone.
+
 If you do not have the 32 bytes hex key (64 hexdigits), place the decryption key file (encrypted_backup.key) extracted from Android. If you gave the 32 bytes hex key, simply put the key in the shell.
 
 Now, you should have something like this in the working directory (if you do not have 32 bytes hex key).
@@ -80,12 +100,13 @@ wtsexporter -a -k 432435053b5204b08e5c3823423399aa30ff061435ab89bc4e6713969cdaa5
 ```
 
 ## Working with iOS/iPadOS (iPhone or iPad)
-Do an iPhone/iPad Backup with iTunes first.
+Do an iPhone/iPad Backup with iTunes/Finder first.
 * iPhone backup on Mac: https://support.apple.com/HT211229
 * iPhone backup on Windows: https://support.apple.com/HT212156
 * iPad backup: https://support.apple.com/guide/ipad/ipad9a74df05xx/ipados
 ### Encrypted iOS/iPadOS Backup
-**If you are working on unencrypted iOS/iPadOS backup, skip this**
+> [!NOTE]
+> If you are working on unencrypted iOS/iPadOS backup, skip this.
 
 If you want to work on an encrypted iOS/iPadOS Backup, you should install iphone_backup_decrypt from [KnugiHK/iphone_backup_decrypt](https://github.com/KnugiHK/iphone_backup_decrypt) before you run the extract_iphone_media.py.
 ```sh
@@ -114,30 +135,38 @@ After extracting, you will get these:
 Invoke the wtsexporter with --help option will show you all options available.
 ```sh
 > wtsexporter --help
-usage: wtsexporter [-h] [-a] [-i] [-e EXPORTED] [-w WA] [-m MEDIA] [-b BACKUP] [-o OUTPUT] [-j [JSON]] [-d DB]
-                   [-k KEY] [-t TEMPLATE] [-s] [-c] [--offline OFFLINE] [--size [SIZE]] [--no-html] [--check-update]
-                   [--assume-first-as-me] [--no-avatar] [--import] [--business] [--preserve-timestamp] [--wab WAB]
-                   [--time-offset {-12 to 14}] [--date DATE] [--date-format FORMAT] [--include [phone number ...]]
-                   [--exclude [phone number ...]] [--create-separated-media]
+usage: wtsexporter [-h] [-a] [-i] [-e EXPORTED] [-w WA] [-m MEDIA] [-b BACKUP] [-o OUTPUT] [-j [JSON]]
+                   [--avoid-encoding-json] [--pretty-print-json [PRETTY_PRINT_JSON]] [-d DB] [-k KEY] [-t TEMPLATE]
+                   [-s] [-c] [--offline OFFLINE] [--size [SIZE]] [--no-html] [--check-update] [--assume-first-as-me]
+                   [--no-avatar] [--import] [--business] [--wab WAB] [--time-offset {-12 to 14}] [--date DATE]
+                   [--date-format FORMAT] [--include [phone number ...]] [--exclude [phone number ...]]
+                   [--dont-filter-empty] [--per-chat] [--create-separated-media]
+                   [--decrypt-chunk-size DECRYPT_CHUNK_SIZE] [--enrich-from-vcards ENRICH_FROM_VCARDS]
+                   [--default-country-code DEFAULT_CONTRY_CODE] [--txt [TEXT_FORMAT]] [--experimental-new-theme]
+                   [--call-db [CALL_DB_IOS]] [--headline HEADLINE]
 
-A customizable Android and iPhone WhatsApp database parser that will give you the history of your WhatsApp
+A customizable Android and iOS/iPadOS WhatsApp database parser that will give you the history of your WhatsApp
 conversations in HTML and JSON. Android Backup Crypt12, Crypt14 and Crypt15 supported.
 
 options:
   -h, --help            show this help message and exit
   -a, --android         Define the target as Android
-  -i, --ios, --iphone   Define the target as iPhone/iPad
+  -i, --ios,            Define the target as iPhone/iPad
   -e EXPORTED, --exported EXPORTED
                         Define the target as exported chat file and specify the path to the file
   -w WA, --wa WA        Path to contact database (default: wa.db/ContactsV2.sqlite)
   -m MEDIA, --media MEDIA
                         Path to WhatsApp media folder (default: WhatsApp)
   -b BACKUP, --backup BACKUP
-                        Path to Android (must be used together with -k)/iPhone WhatsApp backup
+                        Path to Android (must be used together with -k)/iOS WhatsApp backup
   -o OUTPUT, --output OUTPUT
                         Output to specific directory (default: result)
   -j [JSON], --json [JSON]
                         Save the result to a single JSON file (default if present: result.json)
+  --avoid-encoding-json
+                        Don't encode non-ascii characters in the output JSON files
+  --pretty-print-json [PRETTY_PRINT_JSON]
+                        Pretty print the output JSON.
   -d DB, --db DB        Path to database file (default: msgstore.db/7c7fba66680ef796b916b067077cc246adacf01d)
   -k KEY, --key KEY     Path to key file
   -t TEMPLATE, --template TEMPLATE
@@ -153,7 +182,6 @@ options:
   --no-avatar           Do not render avatar in HTML output
   --import              Import JSON file and convert to HTML output
   --business            Use Whatsapp Business default files (iOS only)
-  --preserve-timestamp  Preserve the modification timestamp of the extracted files (iOS only)
   --wab WAB, --wa-backup WAB
                         Path to contact database in crypt15 format
   --time-offset {-12 to 14}
@@ -164,11 +192,29 @@ options:
                         Include chats that match the supplied phone number
   --exclude [phone number ...]
                         Exclude chats that match the supplied phone number
+  --dont-filter-empty   By default, the exporter will not render chats with no valid message. Setting this flag will
+                        cause the exporter to render those. This is useful if chat(s) are missing from the output
+  --per-chat            Output the JSON file per chat
   --create-separated-media
-                        Create a copy of the media seperated per chat in <MEDIA>/separated/ directory 
-                        (Android only)
+                        Create a copy of the media seperated per chat in <MEDIA>/separated/ directory
+  --decrypt-chunk-size DECRYPT_CHUNK_SIZE
+                        Specify the chunk size for decrypting iOS backup, which may affect the decryption speed.
+  --enrich-from-vcards ENRICH_FROM_VCARDS
+                        Path to an exported vcf file from Google contacts export. Add names missing from WhatsApp's
+                        default database
+  --default-country-code DEFAULT_CONTRY_CODE
+                        Use with --enrich-from-vcards. When numbers in the vcf file does not have a country code, this
+                        will be used. 1 is for US, 66 for Thailand etc. Most likely use the number of your own country
+  --txt [TEXT_FORMAT]   Export chats in text format similar to what WhatsApp officially provided (default if present:
+                        result/)
+  --experimental-new-theme
+                        Use the newly designed WhatsApp-alike theme
+  --call-db [CALL_DB_IOS]
+                        Path to call database (default: 1b432994e958845fffe8e2f190f26d1511534088) iOS only
+  --headline HEADLINE   The custom headline for the HTML output. Use '??' as a placeholder for the chat name
 
-WhatsApp Chat Exporter: 0.10.0 Licensed with MIT
+WhatsApp Chat Exporter: 0.11.0 Licensed with MIT. See https://wts.knugi.dev/docs?dest=osl for all open source
+licenses.
 ```
 
 # To do
@@ -179,6 +225,8 @@ This is a MIT licensed project.
 
 The Telegram Desktop's export is the reference for whatsapp.html in this repo.
 
-bplist.py was released by Vladimir "Farcaller" Pouzanov under MIT license.
+`bplist.py` was released by Vladimir "Farcaller" Pouzanov under MIT license.
+
+Please also refer to any files prefixed with `LICENSE` to obtain copies of the various licenses.
 
 WhatsApp Chat Exporter is not affiliated, associated, authorized, endorsed by, or in any way officially connected with the WhatsApp LLC, or any of its subsidiaries or its affiliates. The official WhatsApp LLC website can be found at https://www.whatsapp.com/.

Whatsapp_Chat_Exporter/__init__.py

@@ -1,3 +0,0 @@
-#!/usr/bin/python3
-
-__version__ = "0.10.0"

Whatsapp_Chat_Exporter/__main__.py

@@ -7,18 +7,22 @@ import shutil
 import json
 import string
 import glob
+try:
+    import vobject
+except ModuleNotFoundError:
+    vcards_deps_installed = False
+else:
+    from Whatsapp_Chat_Exporter.vcards_contacts import ContactsFromVCards
+    vcards_deps_installed = True
 from Whatsapp_Chat_Exporter import exported_handler, android_handler
 from Whatsapp_Chat_Exporter import ios_handler, ios_media_handler
 from Whatsapp_Chat_Exporter.data_model import ChatStore
-from Whatsapp_Chat_Exporter.utility import APPLE_TIME, Crypt, DbType
-from Whatsapp_Chat_Exporter.utility import check_update, import_from_json
+from Whatsapp_Chat_Exporter.utility import APPLE_TIME, Crypt, DbType, readable_to_bytes, check_update
+from Whatsapp_Chat_Exporter.utility import import_from_json, sanitize_filename, bytes_to_readable
 from argparse import ArgumentParser, SUPPRESS
 from datetime import datetime
 from sys import exit
-try:
-    from .__init__ import __version__
-except ImportError:
-    from Whatsapp_Chat_Exporter.__init__ import __version__
+import importlib.metadata
 
 
 def main():
@@ -26,7 +30,7 @@ def main():
         description = 'A customizable Android and iOS/iPadOS WhatsApp database parser that '
                       'will give you the history of your WhatsApp conversations in HTML '
                       'and JSON. Android Backup Crypt12, Crypt14 and Crypt15 supported.',
-        epilog = f'WhatsApp Chat Exporter: {__version__} Licensed with MIT. See'
+        epilog = f'WhatsApp Chat Exporter: {importlib.metadata.version("whatsapp_chat_exporter")} Licensed with MIT. See '
                   'https://wts.knugi.dev/docs?dest=osl for all open source licenses.'
     )
     parser.add_argument(
@@ -39,7 +43,6 @@ def main():
     parser.add_argument(
         '-i',
         '--ios',
-        '--iphone',
         dest='ios',
         default=False,
         action='store_true',
@@ -85,6 +88,20 @@ def main():
         type=str,
         const="result.json",
         help="Save the result to a single JSON file (default if present: result.json)")
+    parser.add_argument(
+        '--avoid-encoding-json',
+        dest='avoid_encoding_json',
+        default=False,
+        action='store_true',
+        help="Don't encode non-ascii characters in the output JSON files")
+    parser.add_argument(
+        '--pretty-print-json',
+        dest='pretty_print_json',
+        default=None,
+        nargs='?',
+        const=2,
+        type=int,
+        help="Pretty print the output JSON.")
     parser.add_argument(
         '-d',
         '--db',
@@ -141,7 +158,6 @@ def main():
         "--split",
         dest="size",
         nargs='?',
-        type=int,
         const=0,
         default=None,
         help="Maximum (rough) size of a single output file in bytes, 0 for auto"
@@ -188,13 +204,6 @@ def main():
         action='store_true',
         help="Use Whatsapp Business default files (iOS only)"
     )
-    parser.add_argument(
-        "--preserve-timestamp",
-        dest="preserve_timestamp",
-        default=False,
-        action='store_true',
-        help="Preserve the modification timestamp of the extracted files (iOS only)"
-    )
     parser.add_argument(
         "--wab",
         "--wa-backup",
@@ -239,6 +248,15 @@ def main():
         metavar="phone number",
         help="Exclude chats that match the supplied phone number"
     )
+    parser.add_argument(
+        "--dont-filter-empty",
+        dest="filter_empty",
+        default=True,
+        action='store_false',
+        help=("By default, the exporter will not render chats with no valid message. "
+              "Setting this flag will cause the exporter to render those. "
+              "This is useful if chat(s) are missing from the output")
+    )
     parser.add_argument(
         "--per-chat",
         dest="json_per_chat",
@@ -253,6 +271,57 @@ def main():
         action='store_true',
         help="Create a copy of the media seperated per chat in <MEDIA>/separated/ directory"
     )
+    parser.add_argument(
+        "--decrypt-chunk-size",
+        dest="decrypt_chunk_size",
+        default=1 * 1024 * 1024,
+        type=int,
+        help="Specify the chunk size for decrypting iOS backup, which may affect the decryption speed."
+    )
+    parser.add_argument(
+        "--enrich-from-vcards",
+        dest="enrich_from_vcards",
+        default=None,
+        help="Path to an exported vcf file from Google contacts export. Add names missing from WhatsApp's default database"
+    )
+    parser.add_argument(
+        "--default-country-code",
+        dest="default_contry_code",
+        default=None,
+        help="Use with --enrich-from-vcards. When numbers in the vcf file does not have a country code, this will be used. 1 is for US, 66 for Thailand etc. Most likely use the number of your own country"
+    )
+    parser.add_argument(
+        "--txt",
+        dest="text_format",
+        nargs='?',
+        default=None,
+        type=str,
+        const="result",
+        help="Export chats in text format similar to what WhatsApp officially provided (default if present: result/)"
+    )
+    parser.add_argument(
+        "--experimental-new-theme",
+        dest="whatsapp_theme",
+        default=False,
+        action='store_true',
+        help="Use the newly designed WhatsApp-alike theme"
+    )
+    parser.add_argument(
+        "--call-db",
+        dest="call_db_ios",
+        nargs='?',
+        default=None,
+        type=str,
+        const="1b432994e958845fffe8e2f190f26d1511534088",
+        help="Path to call database (default: 1b432994e958845fffe8e2f190f26d1511534088) iOS only"
+    )
+    parser.add_argument(
+        "--headline",
+        dest="headline",
+        default="Chat history with ??",
+        help="The custom headline for the HTML output. Use '??' as a placeholder for the chat name"
+    )
+
     args = parser.parse_args()
 
     # Check for updates
@@ -264,19 +333,28 @@ def main():
         parser.error("You must define only one device type.")
     if not args.android and not args.ios and not args.exported and not args.import_json:
         parser.error("You must define the device type.")
-    if args.no_html and not args.json:
-        parser.error("You must either specify a JSON output file or enable HTML output.")
+    if args.no_html and not args.json and not args.text_format:
+        parser.error("You must either specify a JSON output file, text file output directory or enable HTML output.")
     if args.import_json and (args.android or args.ios or args.exported or args.no_html):
-        parser.error("You can only use --import with -j and without --no-html.")
+        parser.error("You can only use --import with -j and without --no-html, -a, -i, -e.")
     elif args.import_json and not os.path.isfile(args.json):
         parser.error("JSON file not found.")
     if args.android and args.business:
         parser.error("WhatsApp Business is only available on iOS for now.")
+    if "??" not in args.headline:
+        parser.error("--headline must contain '??' for replacement.")
     if args.json_per_chat and (
         (args.json[-5:] != ".json" and os.path.isfile(args.json)) or \
         (args.json[-5:] == ".json" and os.path.isfile(args.json[:-5]))
     ):
         parser.error("When --per-chat is enabled, the destination of --json must be a directory.")
+    if args.enrich_from_vcards is not None and args.default_contry_code is None:
+        parser.error("When --enrich-from-vcards is provided, you must also set --default-country-code")
+    if args.size is not None and not isinstance(args.size, int) and not args.size.isnumeric():
+        try:
+            args.size = readable_to_bytes(args.size)
+        except ValueError:
+            parser.error("The value for --split must be ended in pure bytes or with a proper unit (e.g., 1048576 or 1MB)")
     if args.filter_date is not None:
         if " - " in args.filter_date:
             start, end = args.filter_date.split(" - ")
@@ -306,6 +384,8 @@ def main():
                     args.filter_date = f"<= {_timestamp - APPLE_TIME}"
             else:
                 parser.error("Unsupported date format. See https://wts.knugi.dev/docs?dest=date")
+    if args.whatsapp_theme:
+        args.template = "whatsapp_new.html"
     if args.filter_chat_include is not None and args.filter_chat_exclude is not None:
         parser.error("Chat inclusion and exclusion filters cannot be used together.")
     if args.filter_chat_include is not None:
@@ -320,6 +400,16 @@ def main():
 
     data = {}
 
+    if args.enrich_from_vcards is not None:
+        if not vcards_deps_installed:
+            parser.error(
+                "You don't have the dependency to enrich contacts with vCard.\n"
+                "Read more on how to deal with enriching contacts:\n"
+                "https://github.com/KnugiHK/Whatsapp-Chat-Exporter/blob/main/README.md#usage"
+            )
+        contact_store = ContactsFromVCards()
+        contact_store.load_vcf_file(args.enrich_from_vcards, args.default_contry_code)
+
     if args.android:
         contacts = android_handler.contacts
         messages = android_handler.messages
@@ -384,12 +474,6 @@ def main():
                 db.row_factory = sqlite3.Row
                 contacts(db, data)
     elif args.ios:
-        import sys
-        if "--iphone" in sys.argv:
-            print(
-                "WARNING: The --iphone flag is deprecated and will"
-                "be removed in the future. Use --ios instead."
-            )
         contacts = ios_handler.contacts
         messages = ios_handler.messages
         media = ios_handler.media
@@ -403,7 +487,7 @@ def main():
             args.media = identifiers.DOMAIN
         if args.backup is not None:
             if not os.path.isdir(args.media):
-                ios_media_handler.extract_media(args.backup, identifiers, args.preserve_timestamp)
+                ios_media_handler.extract_media(args.backup, identifiers, args.decrypt_chunk_size)
             else:
                 print("WhatsApp directory already exists, skipping WhatsApp file extraction.")
         if args.db is None:
@@ -423,12 +507,19 @@ def main():
         if os.path.isfile(msg_db):
             with sqlite3.connect(msg_db) as db:
                 db.row_factory = sqlite3.Row
-                messages(db, data, args.media, args.timezone_offset, args.filter_date, filter_chat)
-                media(db, data, args.media, args.filter_date, filter_chat, args.separate_media)
-                vcard(db, data, args.media, args.filter_date, filter_chat)
+                messages(db, data, args.media, args.timezone_offset, args.filter_date, filter_chat, args.filter_empty)
+                media(db, data, args.media, args.filter_date, filter_chat, args.filter_empty, args.separate_media)
+                vcard(db, data, args.media, args.filter_date, filter_chat, args.filter_empty)
                 if args.android:
                     android_handler.calls(db, data, args.timezone_offset, filter_chat)
+                elif args.ios and args.call_db_ios is not None:
+                    with sqlite3.connect(args.call_db_ios) as cdb:
+                        cdb.row_factory = sqlite3.Row
+                        ios_handler.calls(cdb, data, args.timezone_offset, filter_chat)
             if not args.no_html:
+                if args.enrich_from_vcards is not None and not contact_store.is_empty():
+                    contact_store.enrich_from_vcards(data)
+
                 create_html(
                     data,
                     args.output,
@@ -436,7 +527,9 @@ def main():
                     args.embedded,
                     args.offline,
                     args.size,
-                    args.no_avatar
+                    args.no_avatar,
+                    args.whatsapp_theme,
+                    args.headline
                 )
         else:
             print(
@@ -471,7 +564,10 @@ def main():
                 args.template,
                 args.embedded,
                 args.offline,
-                args.size
+                args.size,
+                args.no_avatar,
+                args.whatsapp_theme,
+                args.headline
             )
         for file in glob.glob(r'*.*'):
             shutil.copy(file, args.output)
@@ -483,16 +579,31 @@ def main():
             args.template,
             args.embedded,
             args.offline,
-            args.size
+            args.size,
+            args.no_avatar,
+            args.whatsapp_theme,
+            args.headline
         )
 
+    if args.text_format:
+        print("Writing text file...")
+        android_handler.create_txt(data, args.text_format)
+
     if args.json and not args.import_json:
+        if args.enrich_from_vcards is not None and not contact_store.is_empty():
+            contact_store.enrich_from_vcards(data)
+
         if isinstance(data[next(iter(data))], ChatStore):
             data = {jik: chat.to_json() for jik, chat in data.items()}
+            
         if not args.json_per_chat:
             with open(args.json, "w") as f:
-                data = json.dumps(data)
-                print(f"\nWriting JSON file...({int(len(data)/1024/1024)}MB)")
+                data = json.dumps(
+                    data,
+                    ensure_ascii=not args.avoid_encoding_json,
+                    indent=args.pretty_print_json
+                )
+                print(f"\nWriting JSON file...({bytes_to_readable(len(data))})")
                 f.write(data)
         else:
             if args.json[-5:] == ".json":
@@ -505,8 +616,9 @@ def main():
                     contact = data[jik]["name"].replace('/', '')
                 else:
                     contact = jik.replace('+', '')
-                with open(f"{args.json}/{contact}.json", "w") as f:
-                    f.write(json.dumps(data[jik]))
+                with open(f"{args.json}/{sanitize_filename(contact)}.json", "w") as f:
+                    file_content_to_write = json.dumps({jik: data[jik]}, ensure_ascii=not args.avoid_encoding_json, indent=2 if args.pretty_print_json else None)
+                    f.write(file_content_to_write)
                     print(f"Writing JSON file...({index + 1}/{total})", end="\r")
             print()
     else:

Whatsapp_Chat_Exporter/android_handler.py

@@ -10,11 +10,12 @@ from mimetypes import MimeTypes
 from markupsafe import escape as htmle
 from hashlib import sha256
 from base64 import b64decode, b64encode
+from datetime import datetime
 from Whatsapp_Chat_Exporter.data_model import ChatStore, Message
-from Whatsapp_Chat_Exporter.utility import MAX_SIZE, ROW_SIZE, DbType, determine_metadata, JidType
+from Whatsapp_Chat_Exporter.utility import CURRENT_TZ_OFFSET, MAX_SIZE, ROW_SIZE, DbType, convert_time_unit, determine_metadata, get_cond_for_empty
 from Whatsapp_Chat_Exporter.utility import rendering, Crypt, Device, get_file_name, setup_template
 from Whatsapp_Chat_Exporter.utility import brute_force_offset, CRYPT14_OFFSETS, get_status_location
-from Whatsapp_Chat_Exporter.utility import get_chat_condition, slugify
+from Whatsapp_Chat_Exporter.utility import get_chat_condition, slugify, bytes_to_readable, JidType
 
 try:
     import zlib
@@ -157,7 +158,11 @@ def contacts(db, data):
     c = db.cursor()
     c.execute("""SELECT count() FROM wa_contacts""")
     total_row_number = c.fetchone()[0]
-    print(f"Processing contacts...({total_row_number})")
+    if total_row_number == 0:
+        print("No contacts profiles found in the default database, consider using --enrich-from-vcards for adopting names from exported contacts from Google")
+        return False
+    else:
+        print(f"Processing contacts...({total_row_number})")
 
     c.execute("""SELECT jid, COALESCE(display_name, wa_name) as display_name, status FROM wa_contacts; """)
     row = c.fetchone()
@@ -168,16 +173,21 @@ def contacts(db, data):
         row = c.fetchone()
 
 
-def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
+def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat, filter_empty):
     # Get message history
     c = db.cursor()
     try:
         c.execute(f"""SELECT count()
                       FROM messages
+                        INNER JOIN jid
+                            ON messages.key_remote_jid = jid.raw_string
+                        LEFT JOIN chat
+                            ON chat.jid_row_id = jid._id
                       WHERE 1=1
+                        {get_cond_for_empty(filter_empty, "messages.key_remote_jid", "messages.needs_push")}
                         {f'AND timestamp {filter_date}' if filter_date is not None else ''}
-                        {get_chat_condition(filter_chat[0], True, "messages.key_remote_jid")}
-                        {get_chat_condition(filter_chat[1], False, "messages.key_remote_jid")}""")
+                        {get_chat_condition(filter_chat[0], True, ["messages.key_remote_jid", "messages.remote_resource"], "jid", "android")}
+                        {get_chat_condition(filter_chat[1], False, ["messages.key_remote_jid", "messages.remote_resource"], "jid", "android")}""")
 
     except sqlite3.OperationalError:
         c.execute(f"""SELECT count()
@@ -186,10 +196,13 @@ def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
                             ON chat._id = message.chat_row_id
                         INNER JOIN jid
                             ON jid._id = chat.jid_row_id
+                        LEFT JOIN jid jid_group
+                            ON jid_group._id = message.sender_jid_row_id
                       WHERE 1=1
+                        {get_cond_for_empty(filter_empty, "jid.raw_string", "broadcast")}
                         {f'AND timestamp {filter_date}' if filter_date is not None else ''}
-                        {get_chat_condition(filter_chat[0], True, "jid.raw_string")}
-                        {get_chat_condition(filter_chat[1], False, "jid.raw_string")}""")
+                        {get_chat_condition(filter_chat[0], True, ["jid.raw_string", "jid_group.raw_string"], "jid", "android")}
+                        {get_chat_condition(filter_chat[1], False, ["jid.raw_string", "jid_group.raw_string"], "jid", "android")}""")
     total_row_number = c.fetchone()[0]
     print(f"Processing messages...(0/{total_row_number})", end="\r")
 
@@ -244,9 +257,10 @@ def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
                         LEFT JOIN receipt_user
                             ON receipt_user.message_row_id = messages._id
                     WHERE messages.key_remote_jid <> '-1'
+                        {get_cond_for_empty(filter_empty, "messages.key_remote_jid", "messages.needs_push")}
                         {f'AND messages.timestamp {filter_date}' if filter_date is not None else ''}
-                        {get_chat_condition(filter_chat[0], True, "messages.key_remote_jid")}
-                        {get_chat_condition(filter_chat[1], False, "messages.key_remote_jid")}
+                        {get_chat_condition(filter_chat[0], True, ["messages.key_remote_jid", "messages.remote_resource"], "jid_global", "android")}
+                        {get_chat_condition(filter_chat[1], False, ["messages.key_remote_jid", "messages.remote_resource"], "jid_global", "android")}
                     GROUP BY messages._id
                     ORDER BY messages.timestamp ASC;"""
         )
@@ -312,9 +326,10 @@ def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
                         LEFT JOIN receipt_user
                             ON receipt_user.message_row_id = message._id
                     WHERE key_remote_jid <> '-1'
+                        {get_cond_for_empty(filter_empty, "key_remote_jid", "broadcast")}
                         {f'AND message.timestamp {filter_date}' if filter_date is not None else ''}
-                        {get_chat_condition(filter_chat[0], True, "key_remote_jid")}
-                        {get_chat_condition(filter_chat[1], False, "key_remote_jid")}
+                        {get_chat_condition(filter_chat[0], True, ["key_remote_jid", "jid_group.raw_string"], "jid_global", "android")}
+                        {get_chat_condition(filter_chat[1], False, ["key_remote_jid", "jid_group.raw_string"], "jid_global", "android")}
                     GROUP BY message._id;"""
             )
         except Exception as e:
@@ -345,7 +360,8 @@ def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
             timestamp=content["timestamp"],
             time=content["timestamp"],
             key_id=content["key_id"],
-            timezone_offset=timezone_offset
+            timezone_offset=timezone_offset if timezone_offset else CURRENT_TZ_OFFSET,
+            message_type=content["media_wa_type"]
         )
         if isinstance(content["data"], bytes):
             message.data = ("The message is binary data and its base64 is "
@@ -445,9 +461,9 @@ def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
                         msg = content["data"]
                         if msg is not None:
                             if "\r\n" in msg:
-                                msg = msg.replace("\r\n", "<br>")
+                                msg = msg.replace("\r\n", " <br>")
                             if "\n" in msg:
-                                msg = msg.replace("\n", "<br>")
+                                msg = msg.replace("\n", " <br>")
             else:
                 if content["status"] == 0 and content["edit_version"] == 7 or table_message and content["media_wa_type"] == 15:
                     msg = "Message deleted"
@@ -460,9 +476,9 @@ def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
                         msg = content["data"]
                         if msg is not None:
                             if "\r\n" in msg:
-                                msg = msg.replace("\r\n", "<br>")
+                                msg = msg.replace("\r\n", " <br>")
                             if "\n" in msg:
-                                msg = msg.replace("\n", "<br>")
+                                msg = msg.replace("\n", " <br>")
             message.data = msg
 
         data[content["key_remote_jid"]].add_message(content["_id"], message)
@@ -479,7 +495,7 @@ def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
     print(f"Processing messages...({total_row_number}/{total_row_number})", end="\r")
 
 
-def media(db, data, media_folder, filter_date, filter_chat, separate_media=True):
+def media(db, data, media_folder, filter_date, filter_chat, filter_empty, separate_media=True):
     # Get media
     c = db.cursor()
     try:
@@ -487,10 +503,15 @@ def media(db, data, media_folder, filter_date, filter_chat, separate_media=True)
                     FROM message_media
                         INNER JOIN messages
                             ON message_media.message_row_id = messages._id
+                        INNER JOIN jid
+                            ON messages.key_remote_jid = jid.raw_string
+                        LEFT JOIN chat
+                            ON chat.jid_row_id = jid._id
                     WHERE 1=1  
+                        {get_cond_for_empty(filter_empty, "key_remote_jid", "messages.needs_push")}
                         {f'AND messages.timestamp {filter_date}' if filter_date is not None else ''}
-                        {get_chat_condition(filter_chat[0], True, "messages.key_remote_jid")}
-                        {get_chat_condition(filter_chat[1], False, "messages.key_remote_jid")}""")
+                        {get_chat_condition(filter_chat[0], True, ["messages.key_remote_jid", "remote_resource"], "jid", "android")}
+                        {get_chat_condition(filter_chat[1], False, ["messages.key_remote_jid", "remote_resource"], "jid", "android")}""")
     except sqlite3.OperationalError:
         c.execute(f"""SELECT count()
                     FROM message_media
@@ -500,10 +521,13 @@ def media(db, data, media_folder, filter_date, filter_chat, separate_media=True)
                             ON chat._id = message.chat_row_id
                         INNER JOIN jid
                             ON jid._id = chat.jid_row_id
+                        LEFT JOIN jid jid_group
+                            ON jid_group._id = message.sender_jid_row_id
                     WHERE 1=1    
+                        {get_cond_for_empty(filter_empty, "jid.raw_string", "broadcast")}
                         {f'AND message.timestamp {filter_date}' if filter_date is not None else ''}
-                        {get_chat_condition(filter_chat[0], True, "jid.raw_string")}
-                        {get_chat_condition(filter_chat[1], False, "jid.raw_string")}""")
+                        {get_chat_condition(filter_chat[0], True, ["jid.raw_string", "jid_group.raw_string"], "jid", "android")}
+                        {get_chat_condition(filter_chat[1], False, ["jid.raw_string", "jid_group.raw_string"], "jid", "android")}""")
     total_row_number = c.fetchone()[0]
     print(f"\nProcessing media...(0/{total_row_number})", end="\r")
     i = 0
@@ -523,10 +547,13 @@ def media(db, data, media_folder, filter_date, filter_chat, separate_media=True)
 						ON message_media.file_hash = media_hash_thumbnail.media_hash
                     INNER JOIN jid
                         ON messages.key_remote_jid = jid.raw_string
+                    LEFT JOIN chat
+                        ON chat.jid_row_id = jid._id
                 WHERE jid.type <> 7
+                    {get_cond_for_empty(filter_empty, "key_remote_jid", "broadcast")}
                     {f'AND messages.timestamp {filter_date}' if filter_date is not None else ''}
-                    {get_chat_condition(filter_chat[0], True, "messages.key_remote_jid")}
-                    {get_chat_condition(filter_chat[1], False, "messages.key_remote_jid")}
+                    {get_chat_condition(filter_chat[0], True, ["messages.key_remote_jid", "remote_resource"], "jid", "android")}
+                    {get_chat_condition(filter_chat[1], False, ["messages.key_remote_jid", "remote_resource"], "jid", "android")}
                 ORDER BY messages.key_remote_jid ASC"""
         )
     except sqlite3.OperationalError:
@@ -547,10 +574,13 @@ def media(db, data, media_folder, filter_date, filter_chat, separate_media=True)
                         ON jid._id = chat.jid_row_id
                     LEFT JOIN media_hash_thumbnail
 						ON message_media.file_hash = media_hash_thumbnail.media_hash
+                    LEFT JOIN jid jid_group
+                        ON jid_group._id = message.sender_jid_row_id
                 WHERE jid.type <> 7
+                    {get_cond_for_empty(filter_empty, "key_remote_jid", "broadcast")}
                     {f'AND message.timestamp {filter_date}' if filter_date is not None else ''}
-                    {get_chat_condition(filter_chat[0], True, "key_remote_jid")}
-                    {get_chat_condition(filter_chat[1], False, "key_remote_jid")}
+                    {get_chat_condition(filter_chat[0], True, ["key_remote_jid", "jid_group.raw_string"], "jid", "android")}
+                    {get_chat_condition(filter_chat[1], False, ["key_remote_jid", "jid_group.raw_string"], "jid", "android")}
                 ORDER BY jid.raw_string ASC"""
         )
     content = c.fetchone()
@@ -581,17 +611,6 @@ def media(db, data, media_folder, filter_date, filter_chat, separate_media=True)
                 shutil.copy2(file_path, new_path)
                 message.data = new_path
         else:
-            if False: # Block execution
-                try:
-                    r = requests.get(content["message_url"])
-                    if r.status_code != 200:
-                        raise RuntimeError()
-                except:
-                    message.data = "The media is missing"
-                    message.mime = "media"
-                    message.meta = True
-                else:
-                    ...
             message.data = "The media is missing"
             message.mime = "media"
             message.meta = True
@@ -609,7 +628,7 @@ def media(db, data, media_folder, filter_date, filter_chat, separate_media=True)
         f"Processing media...({total_row_number}/{total_row_number})", end="\r")
 
 
-def vcard(db, data, media_folder, filter_date, filter_chat):
+def vcard(db, data, media_folder, filter_date, filter_chat, filter_empty):
     c = db.cursor()
     try:
         c.execute(f"""SELECT message_row_id,
@@ -619,10 +638,15 @@ def vcard(db, data, media_folder, filter_date, filter_chat):
                  FROM messages_vcards
                     INNER JOIN messages
                         ON messages_vcards.message_row_id = messages._id
+                    INNER JOIN jid
+                        ON messages.key_remote_jid = jid.raw_string
+                    LEFT JOIN chat
+                        ON chat.jid_row_id = jid._id
                  WHERE 1=1
+                    {get_cond_for_empty(filter_empty, "key_remote_jid", "messages.needs_push")}
                     {f'AND messages.timestamp {filter_date}' if filter_date is not None else ''}
-                    {get_chat_condition(filter_chat[0], True, "messages.key_remote_jid")}
-                    {get_chat_condition(filter_chat[1], False, "messages.key_remote_jid")}
+                    {get_chat_condition(filter_chat[0], True, ["messages.key_remote_jid", "remote_resource"], "jid", "android")}
+                    {get_chat_condition(filter_chat[1], False, ["messages.key_remote_jid", "remote_resource"], "jid", "android")}
                  ORDER BY messages.key_remote_jid ASC;"""
         )
     except sqlite3.OperationalError:
@@ -637,10 +661,13 @@ def vcard(db, data, media_folder, filter_date, filter_chat):
                         ON chat._id = message.chat_row_id
                     INNER JOIN jid
                         ON jid._id = chat.jid_row_id
+                    LEFT JOIN jid jid_group
+                        ON jid_group._id = message.sender_jid_row_id
                 WHERE 1=1
+                    {get_cond_for_empty(filter_empty, "key_remote_jid", "broadcast")}
                     {f'AND message.timestamp {filter_date}' if filter_date is not None else ''}
-                    {get_chat_condition(filter_chat[0], True, "key_remote_jid")}
-                    {get_chat_condition(filter_chat[1], False, "key_remote_jid")}
+                    {get_chat_condition(filter_chat[0], True, ["key_remote_jid", "jid_group.raw_string"], "jid", "android")}
+                    {get_chat_condition(filter_chat[1], False, ["key_remote_jid", "jid_group.raw_string"], "jid", "android")}
                  ORDER BY message.chat_row_id ASC;"""
         )
 
@@ -676,8 +703,8 @@ def calls(db, data, timezone_offset, filter_chat):
                     LEFT JOIN chat
                         ON call_log.jid_row_id = chat.jid_row_id
                 WHERE 1=1
-                    {get_chat_condition(filter_chat[0], True, "jid.raw_string")}
-                    {get_chat_condition(filter_chat[1], False, "jid.raw_string")}""")
+                    {get_chat_condition(filter_chat[0], True, ["jid.raw_string"])}
+                    {get_chat_condition(filter_chat[1], False, ["jid.raw_string"])}""")
     total_row_number = c.fetchone()[0]
     if total_row_number == 0:
         return
@@ -698,8 +725,8 @@ def calls(db, data, timezone_offset, filter_chat):
                     LEFT JOIN chat
                         ON call_log.jid_row_id = chat.jid_row_id
                 WHERE 1=1
-                    {get_chat_condition(filter_chat[0], True, "jid.raw_string")}
-                    {get_chat_condition(filter_chat[1], False, "jid.raw_string")}"""
+                    {get_chat_condition(filter_chat[0], True, ["jid.raw_string"])}
+                    {get_chat_condition(filter_chat[1], False, ["jid.raw_string"])}"""
     )
     chat = ChatStore(Device.ANDROID, "WhatsApp Calls")
     content = c.fetchone()
@@ -709,7 +736,7 @@ def calls(db, data, timezone_offset, filter_chat):
             timestamp=content["timestamp"],
             time=content["timestamp"],
             key_id=content["call_id"],
-            timezone_offset=timezone_offset
+            timezone_offset=timezone_offset if timezone_offset else CURRENT_TZ_OFFSET
         )
         _jid = content["raw_string"]
         name = data[_jid].name if _jid in data else content["chat_subject"] or None
@@ -724,15 +751,21 @@ def calls(db, data, timezone_offset, filter_chat):
             f"call {'to' if call.from_me else 'from'} "
             f"{call.sender} was "
         )
-        if content['call_result'] == 2:
+        if content['call_result'] in (0, 4, 7):
+            call.data += "cancelled." if call.from_me else "missed."
+        elif content['call_result'] == 2:
             call.data += "not answered." if call.from_me else "missed."
         elif content['call_result'] == 3:
             call.data += "unavailable."
         elif content['call_result'] == 5:
+            call_time = convert_time_unit(content['duration'])
+            call_bytes = bytes_to_readable(content['bytes_transferred'])
             call.data += (
-                f"initiated and lasted for {content['duration']} second(s) "
-                f"with {content['bytes_transferred']} bytes transferred."
+                f"initiated and lasted for {call_time} "
+                f"with {call_bytes} data transferred."
             )
+        else:
+            call.data += "in an unknown state."
         chat.add_message(content["_id"], call)
         content = c.fetchone()
     data["000000000000000"] = chat
@@ -745,9 +778,11 @@ def create_html(
         embedded=False,
         offline_static=False,
         maximum_size=None,
-        no_avatar=False
+        no_avatar=False,
+        experimental=False,
+        headline=None
     ):
-    template = setup_template(template, no_avatar)
+    template = setup_template(template, no_avatar, experimental)
 
     total_row_number = len(data)
     print(f"\nGenerating chats...(0/{total_row_number})", end="\r")
@@ -759,8 +794,6 @@ def create_html(
 
     for current, contact in enumerate(data):
         chat = data[contact]
-        if len(chat.messages) == 0:
-            continue
         safe_file_name, name = get_file_name(contact, chat)
 
         if maximum_size is not None:
@@ -784,13 +817,16 @@ def create_html(
                         render_box,
                         contact,
                         w3css,
-                        f"{safe_file_name}-{current_page + 1}.html",
-                        chat
+                        chat,
+                        headline,
+                        next=f"{safe_file_name}-{current_page + 1}.html",
+                        previous=f"{safe_file_name}-{current_page - 1}.html" if current_page > 1 else False
                     )
                     render_box = [message]
                     current_size = 0
                     current_page += 1
                 else:
+                    render_box.append(message)
                     if message.key_id == last_msg:
                         if current_page == 1:
                             output_file_name = f"{output_folder}/{safe_file_name}.html"
@@ -803,11 +839,11 @@ def create_html(
                             render_box,
                             contact,
                             w3css,
+                            chat,
+                            headline,
                             False,
-                            chat
+                            previous=f"{safe_file_name}-{current_page - 1}.html"
                         )
-                    else:
-                        render_box.append(message)
         else:
             output_file_name = f"{output_folder}/{safe_file_name}.html"
             rendering(
@@ -817,10 +853,46 @@ def create_html(
                 chat.get_messages(),
                 contact,
                 w3css,
-                False,
-                chat
+                chat,
+                headline,
+                False
             )
         if current % 10 == 0:
             print(f"Generating chats...({current}/{total_row_number})", end="\r")
 
     print(f"Generating chats...({total_row_number}/{total_row_number})", end="\r")
+
+
+def create_txt(data, output):
+    os.makedirs(output, exist_ok=True)
+    for jik, chat in data.items():
+        if chat.name is not None:
+            contact = chat.name.replace('/', '')
+        else:
+            contact = jik.replace('+', '')
+        output_file = os.path.join(output, f"{contact}.txt")
+        with open(output_file, "w", encoding="utf8") as f:
+            for message in chat.messages.values():
+                date = datetime.fromtimestamp(message.timestamp).date()
+                if message.meta and message.mime != "media":
+                    continue  # Skip any metadata in text format
+                if message.from_me:
+                    name = "You"
+                else:
+                    name = message.sender if message.sender else contact
+                prefix = f"[{date} {message.time}] {name}: "
+                prefix_length = len(prefix)
+                if message.media and ("/" in message.mime or message.mime == "media"):
+                    if message.data == "The media is missing":
+                        message_text = "<The media is missing>"
+                    else:
+                        message_text = f"<media file in {message.data}>"
+                else:
+                    if message.data is None:
+                        message_text = ""
+                    else:
+                        message_text = message.data.replace('<br>', f'\n{" " * prefix_length}')
+                if message.caption is not None:
+                    message_text += "\n" + ' ' * len(prefix) + message.caption.replace('<br>', f'\n{" " * prefix_length}')
+                f.write(f"{prefix}{message_text}\n")
+

Whatsapp_Chat_Exporter/data_model.py

@@ -34,6 +34,7 @@ class ChatStore():
         self.their_avatar = None
         self.their_avatar_thumb = None
         self.status = None
+        self.media_base = ""
     
     def add_message(self, id, message):
         if not isinstance(message, Message):
@@ -64,7 +65,7 @@ class ChatStore():
 
 
 class Message():
-    def __init__(self, from_me: Union[bool,int], timestamp: int, time: Union[int,float,str], key_id: int, timezone_offset: int = 0):
+    def __init__(self, from_me: Union[bool,int], timestamp: int, time: Union[int,float,str], key_id: int, timezone_offset: int = 0, message_type: int = None):
         self.from_me = bool(from_me)
         self.timestamp = timestamp / 1000 if timestamp > 9999999999 else timestamp
         if isinstance(time, int) or isinstance(time, float):
@@ -80,6 +81,7 @@ class Message():
         self.sender = None
         self.safe = False
         self.mime = None
+        self.message_type = message_type
         # Extra
         self.reply = None
         self.quoted_data = None

Whatsapp_Chat_Exporter/ios_handler.py

@@ -7,7 +7,8 @@ from pathlib import Path
 from mimetypes import MimeTypes
 from markupsafe import escape as htmle
 from Whatsapp_Chat_Exporter.data_model import ChatStore, Message
-from Whatsapp_Chat_Exporter.utility import APPLE_TIME, Device, get_chat_condition, slugify
+from Whatsapp_Chat_Exporter.utility import APPLE_TIME, CURRENT_TZ_OFFSET, get_chat_condition
+from Whatsapp_Chat_Exporter.utility import bytes_to_readable, convert_time_unit, slugify, Device
 
 
 def contacts(db, data):
@@ -26,27 +27,45 @@ def contacts(db, data):
         content = c.fetchone()
 
 
-def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
+def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat, filter_empty):
     c = db.cursor()
+    cursor2 = db.cursor()
     # Get contacts
-    c.execute(f"""SELECT count()
-                FROM ZWACHATSESSION
-                WHERE 1=1
-                    {get_chat_condition(filter_chat[0], True, "ZWACHATSESSION.ZCONTACTJID")}
-                    {get_chat_condition(filter_chat[1], False, "ZWACHATSESSION.ZCONTACTJID")}""")
+    c.execute(
+        f"""SELECT count() 
+            FROM (SELECT DISTINCT ZCONTACTJID,
+                ZPARTNERNAME,
+                ZWAPROFILEPUSHNAME.ZPUSHNAME
+            FROM ZWACHATSESSION
+                INNER JOIN ZWAMESSAGE
+                    ON ZWAMESSAGE.ZCHATSESSION = ZWACHATSESSION.Z_PK
+                LEFT JOIN ZWAPROFILEPUSHNAME
+                    ON ZWACHATSESSION.ZCONTACTJID = ZWAPROFILEPUSHNAME.ZJID
+                LEFT JOIN ZWAGROUPMEMBER
+                        ON ZWAMESSAGE.ZGROUPMEMBER = ZWAGROUPMEMBER.Z_PK
+            WHERE 1=1
+                {get_chat_condition(filter_chat[0], True, ["ZWACHATSESSION.ZCONTACTJID", "ZMEMBERJID"], "ZGROUPINFO", "ios")}
+                {get_chat_condition(filter_chat[1], False, ["ZWACHATSESSION.ZCONTACTJID", "ZMEMBERJID"], "ZGROUPINFO", "ios")}
+            GROUP BY ZCONTACTJID);"""
+    )
     total_row_number = c.fetchone()[0]
     print(f"Processing contacts...({total_row_number})")
 
     c.execute(
-        f"""SELECT ZCONTACTJID,
+        f"""SELECT DISTINCT ZCONTACTJID,
                 ZPARTNERNAME,
-                ZPUSHNAME
-           FROM ZWACHATSESSION
+                ZWAPROFILEPUSHNAME.ZPUSHNAME
+            FROM ZWACHATSESSION
+                INNER JOIN ZWAMESSAGE
+                    ON ZWAMESSAGE.ZCHATSESSION = ZWACHATSESSION.Z_PK
                 LEFT JOIN ZWAPROFILEPUSHNAME
                     ON ZWACHATSESSION.ZCONTACTJID = ZWAPROFILEPUSHNAME.ZJID
-           WHERE 1=1
-                {get_chat_condition(filter_chat[0], True, "ZWACHATSESSION.ZCONTACTJID")}
-                {get_chat_condition(filter_chat[1], False, "ZWACHATSESSION.ZCONTACTJID")};"""
+                LEFT JOIN ZWAGROUPMEMBER
+                        ON ZWAMESSAGE.ZGROUPMEMBER = ZWAGROUPMEMBER.Z_PK
+            WHERE 1=1
+                {get_chat_condition(filter_chat[0], True, ["ZWACHATSESSION.ZCONTACTJID", "ZMEMBERJID"], "ZGROUPINFO", "ios")}
+                {get_chat_condition(filter_chat[1], False, ["ZWACHATSESSION.ZCONTACTJID", "ZMEMBERJID"], "ZGROUPINFO", "ios")}
+            GROUP BY ZCONTACTJID;"""
     )
     content = c.fetchone()
     while content is not None:
@@ -78,10 +97,12 @@ def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
                   FROM ZWAMESSAGE
                         INNER JOIN ZWACHATSESSION
                             ON ZWAMESSAGE.ZCHATSESSION = ZWACHATSESSION.Z_PK
+                        LEFT JOIN ZWAGROUPMEMBER
+                            ON ZWAMESSAGE.ZGROUPMEMBER = ZWAGROUPMEMBER.Z_PK
                   WHERE 1=1
                     {f'AND ZMESSAGEDATE {filter_date}' if filter_date is not None else ''}
-                    {get_chat_condition(filter_chat[0], True, "ZWACHATSESSION.ZCONTACTJID")}
-                    {get_chat_condition(filter_chat[1], False, "ZWACHATSESSION.ZCONTACTJID")}""")
+                    {get_chat_condition(filter_chat[0], True, ["ZWACHATSESSION.ZCONTACTJID", "ZMEMBERJID"], "ZGROUPINFO", "ios")}
+                    {get_chat_condition(filter_chat[1], False, ["ZWACHATSESSION.ZCONTACTJID", "ZMEMBERJID"], "ZGROUPINFO", "ios")}""")
     total_row_number = c.fetchone()[0]
     print(f"Processing messages...(0/{total_row_number})", end="\r")
     c.execute(f"""SELECT ZCONTACTJID,
@@ -103,8 +124,8 @@ def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
                         ON ZWAMESSAGE.ZCHATSESSION = ZWACHATSESSION.Z_PK
                  WHERE 1=1   
                     {f'AND ZMESSAGEDATE {filter_date}' if filter_date is not None else ''}
-                    {get_chat_condition(filter_chat[0], True, "ZCONTACTJID")}
-                    {get_chat_condition(filter_chat[1], False, "ZCONTACTJID")}
+                    {get_chat_condition(filter_chat[0], True, ["ZCONTACTJID", "ZMEMBERJID"], "ZGROUPINFO", "ios")}
+                    {get_chat_condition(filter_chat[1], False, ["ZCONTACTJID", "ZMEMBERJID"], "ZGROUPINFO", "ios")}
                  ORDER BY ZMESSAGEDATE ASC;""")
     i = 0
     content = c.fetchone()
@@ -130,7 +151,8 @@ def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
             timestamp=ts,
             time=ts, # TODO: Could be bug
             key_id=content["ZSTANZAID"][:17],
-            timezone_offset=timezone_offset
+            timezone_offset=timezone_offset if timezone_offset else CURRENT_TZ_OFFSET,
+            message_type=content["ZMESSAGETYPE"]
         )
         invalid = False
         if is_group_message and content["ZISFROMME"] == 0:
@@ -170,7 +192,11 @@ def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
             if content["ZMETADATA"] is not None and content["ZMETADATA"].startswith(b"\x2a\x14"):
                 quoted = content["ZMETADATA"][2:19]
                 message.reply = quoted.decode()
-                message.quoted_data = None # TODO
+                cursor2.execute(f"""SELECT ZTEXT
+                                    FROM ZWAMESSAGE
+                                    WHERE ZSTANZAID LIKE '{message.reply}%'""")
+                quoted_content = cursor2.fetchone()
+                message.quoted_data = quoted_content["ZTEXT"] or quoted_content
             if content["ZMESSAGETYPE"] == 15: # Sticker
                 message.sticker = True
 
@@ -207,7 +233,7 @@ def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
         f"Processing messages...({total_row_number}/{total_row_number})", end="\r")
 
 
-def media(db, data, media_folder, filter_date, filter_chat, separate_media=False):
+def media(db, data, media_folder, filter_date, filter_chat, filter_empty, separate_media=False):
     c = db.cursor()
     # Get media
     c.execute(f"""SELECT count()
@@ -216,10 +242,12 @@ def media(db, data, media_folder, filter_date, filter_chat, separate_media=False
                         ON ZWAMEDIAITEM.ZMESSAGE = ZWAMESSAGE.Z_PK
                     INNER JOIN ZWACHATSESSION
                         ON ZWAMESSAGE.ZCHATSESSION = ZWACHATSESSION.Z_PK
+                    LEFT JOIN ZWAGROUPMEMBER
+                        ON ZWAMESSAGE.ZGROUPMEMBER = ZWAGROUPMEMBER.Z_PK
                   WHERE 1=1
                     {f'AND ZMESSAGEDATE {filter_date}' if filter_date is not None else ''}
-                    {get_chat_condition(filter_chat[0], True, "ZWACHATSESSION.ZCONTACTJID")}
-                    {get_chat_condition(filter_chat[1], False, "ZWACHATSESSION.ZCONTACTJID")}
+                    {get_chat_condition(filter_chat[0], True, ["ZWACHATSESSION.ZCONTACTJID","ZMEMBERJID"], "ZGROUPINFO", "ios")}
+                    {get_chat_condition(filter_chat[1], False, ["ZWACHATSESSION.ZCONTACTJID", "ZMEMBERJID"], "ZGROUPINFO", "ios")}
                 """)
     total_row_number = c.fetchone()[0]
     print(f"\nProcessing media...(0/{total_row_number})", end="\r")
@@ -236,20 +264,25 @@ def media(db, data, media_folder, filter_date, filter_chat, separate_media=False
                         ON ZWAMEDIAITEM.ZMESSAGE = ZWAMESSAGE.Z_PK
                     INNER JOIN ZWACHATSESSION
                         ON ZWAMESSAGE.ZCHATSESSION = ZWACHATSESSION.Z_PK
+                    LEFT JOIN ZWAGROUPMEMBER
+                        ON ZWAMESSAGE.ZGROUPMEMBER = ZWAGROUPMEMBER.Z_PK
                  WHERE ZMEDIALOCALPATH IS NOT NULL
                     {f'AND ZWAMESSAGE.ZMESSAGEDATE {filter_date}' if filter_date is not None else ''}
-                    {get_chat_condition(filter_chat[0], True, "ZCONTACTJID")}
-                    {get_chat_condition(filter_chat[1], False, "ZCONTACTJID")}
+                    {get_chat_condition(filter_chat[0], True, ["ZCONTACTJID", "ZMEMBERJID"], "ZGROUPINFO", "ios")}
+                    {get_chat_condition(filter_chat[1], False, ["ZCONTACTJID", "ZMEMBERJID"], "ZGROUPINFO", "ios")}
                  ORDER BY ZCONTACTJID ASC""")
     content = c.fetchone()
     mime = MimeTypes()
     while content is not None:
         file_path = f"{media_folder}/Message/{content['ZMEDIALOCALPATH']}"
         ZMESSAGE = content["ZMESSAGE"]
-        message = data[content["ZCONTACTJID"]].messages[ZMESSAGE]
+        contact = data[content["ZCONTACTJID"]]
+        message = contact.messages[ZMESSAGE]
         message.media = True
+        if contact.media_base == "":
+            contact.media_base = media_folder + "/"
         if os.path.isfile(file_path):
-            message.data = file_path
+            message.data = '/'.join(file_path.split("/")[1:])
             if content["ZVCARDSTRING"] is None:
                 guess = mime.guess_type(file_path)[0]
                 if guess is not None:
@@ -259,26 +292,15 @@ def media(db, data, media_folder, filter_date, filter_chat, separate_media=False
             else:
                 message.mime = content["ZVCARDSTRING"]
             if separate_media:
-                chat_display_name = slugify(data[content["ZCONTACTJID"]].name or message.sender \
+                chat_display_name = slugify(contact.name or message.sender \
                                             or content["ZCONTACTJID"].split('@')[0], True)
                 current_filename = file_path.split("/")[-1]
                 new_folder = os.path.join(media_folder, "separated", chat_display_name)
                 Path(new_folder).mkdir(parents=True, exist_ok=True)
                 new_path = os.path.join(new_folder, current_filename)
                 shutil.copy2(file_path, new_path)
-                message.data = new_path
+                message.data = '/'.join(new_path.split("\\")[1:])
         else:
-            if False: # Block execution
-                try:
-                    r = requests.get(content["ZMEDIAURL"])
-                    if r.status_code != 200:
-                        raise RuntimeError()
-                except:
-                    message.data = "The media is missing"
-                    message.mime = "media"
-                    message.meta = True
-                else:
-                    ...
             message.data = "The media is missing"
             message.mime = "media"
             message.meta = True
@@ -292,7 +314,7 @@ def media(db, data, media_folder, filter_date, filter_chat, separate_media=False
         f"Processing media...({total_row_number}/{total_row_number})", end="\r")
 
 
-def vcard(db, data, media_folder, filter_date, filter_chat):
+def vcard(db, data, media_folder, filter_date, filter_chat, filter_empty):
     c = db.cursor()
     c.execute(f"""SELECT DISTINCT ZWAVCARDMENTION.ZMEDIAITEM,
                         ZWAMEDIAITEM.ZMESSAGE,
@@ -306,10 +328,12 @@ def vcard(db, data, media_folder, filter_date, filter_chat):
                         ON ZWAMEDIAITEM.ZMESSAGE = ZWAMESSAGE.Z_PK
                     INNER JOIN ZWACHATSESSION
                         ON ZWAMESSAGE.ZCHATSESSION = ZWACHATSESSION.Z_PK
+                    LEFT JOIN ZWAGROUPMEMBER
+                        ON ZWAMESSAGE.ZGROUPMEMBER = ZWAGROUPMEMBER.Z_PK
                  WHERE 1=1
                     {f'AND ZWAMESSAGE.ZMESSAGEDATE {filter_date}' if filter_date is not None else ''}
-                    {get_chat_condition(filter_chat[0], True, "ZCONTACTJID")}
-                    {get_chat_condition(filter_chat[1], False, "ZCONTACTJID")};""")
+                    {get_chat_condition(filter_chat[0], True, ["ZCONTACTJID", "ZMEMBERJID"], "ZGROUPINFO", "ios")}
+                    {get_chat_condition(filter_chat[1], False, ["ZCONTACTJID", "ZMEMBERJID"], "ZGROUPINFO", "ios")};""")
     contents = c.fetchall()
     total_row_number = len(contents)
     print(f"\nProcessing vCards...(0/{total_row_number})", end="\r")
@@ -344,3 +368,73 @@ def vcard(db, data, media_folder, filter_date, filter_chat):
         message.meta = True
         message.safe = True
         print(f"Processing vCards...({index + 1}/{total_row_number})", end="\r")
+
+
+def calls(db, data, timezone_offset, filter_chat):
+    c = db.cursor()
+    c.execute(f"""SELECT count()
+                FROM ZWACDCALLEVENT
+                WHERE 1=1
+                    {get_chat_condition(filter_chat[0], True, ["ZGROUPCALLCREATORUSERJIDSTRING"], None, "ios")}
+                    {get_chat_condition(filter_chat[1], False, ["ZGROUPCALLCREATORUSERJIDSTRING"], None, "ios")}""")
+    total_row_number = c.fetchone()[0]
+    if total_row_number == 0:
+        return
+    print(f"\nProcessing calls...({total_row_number})", end="\r")
+    c.execute(f"""SELECT ZCALLIDSTRING,
+                        ZGROUPCALLCREATORUSERJIDSTRING,
+                        ZGROUPJIDSTRING,
+                        ZDATE,
+                        ZOUTCOME,
+                        ZBYTESRECEIVED + ZBYTESSENT AS bytes_transferred,
+                        ZDURATION,
+                        ZVIDEO,
+                        ZMISSED,
+                        ZINCOMING
+                FROM ZWACDCALLEVENT
+                    INNER JOIN ZWAAGGREGATECALLEVENT
+                        ON ZWACDCALLEVENT.Z1CALLEVENTS = ZWAAGGREGATECALLEVENT.Z_PK
+                WHERE 1=1
+                    {get_chat_condition(filter_chat[0], True, ["ZGROUPCALLCREATORUSERJIDSTRING"], None, "ios")}
+                    {get_chat_condition(filter_chat[1], False, ["ZGROUPCALLCREATORUSERJIDSTRING"], None, "ios")}""")
+    chat = ChatStore(Device.ANDROID, "WhatsApp Calls")
+    content = c.fetchone()
+    while content is not None:
+        ts = APPLE_TIME + int(content["ZDATE"])
+        call = Message(
+            from_me=content["ZINCOMING"] == 0,
+            timestamp=ts,
+            time=ts,
+            key_id=content["ZCALLIDSTRING"],
+            timezone_offset=timezone_offset if timezone_offset else CURRENT_TZ_OFFSET
+        )
+        _jid = content["ZGROUPCALLCREATORUSERJIDSTRING"]
+        name = data[_jid].name if _jid in data else None
+        if _jid is not None and "@" in _jid:
+            fallback = _jid.split('@')[0]
+        else:
+            fallback = None
+        call.sender = name or fallback
+        call.meta = True
+        call.data = (
+            f"A {'group ' if content['ZGROUPJIDSTRING'] is not None else ''}"
+            f"{'video' if content['ZVIDEO'] == 1 else 'voice'} "
+            f"call {'to' if call.from_me else 'from'} "
+            f"{call.sender} was "
+        )
+        if content['ZOUTCOME'] in (1, 4):
+            call.data += "not answered." if call.from_me else "missed."
+        elif content['ZOUTCOME'] == 2:
+            call.data += "failed."
+        elif content['ZOUTCOME'] == 0:
+            call_time = convert_time_unit(int(content['ZDURATION']))
+            call_bytes = bytes_to_readable(content['bytes_transferred'])
+            call.data += (
+                f"initiated and lasted for {call_time} "
+                f"with {call_bytes} data transferred."
+            )
+        else:
+            call.data += "in an unknown state."
+        chat.add_message(call.key_id, call)
+        content = c.fetchone()
+    data["000000000000000"] = chat

Whatsapp_Chat_Exporter/ios_media_handler.py

@@ -3,55 +3,64 @@
 import shutil
 import sqlite3
 import os
-import time
 import getpass
-import threading
 from Whatsapp_Chat_Exporter.utility import WhatsAppIdentifier
+from Whatsapp_Chat_Exporter.bplist import BPListReader
 try:
     from iphone_backup_decrypt import EncryptedBackup, RelativePath
-    from iphone_backup_decrypt import FailedToDecryptError
 except ModuleNotFoundError:
     support_encrypted = False
 else:
     support_encrypted = True
 
 
-def extract_encrypted(base_dir, password, identifiers, bplist_reader=None):
-    backup = EncryptedBackup(backup_directory=base_dir, passphrase=password, cleanup=False, check_same_thread=False)
-    print("Decrypting WhatsApp database...", end="")
+def extract_encrypted(base_dir, password, identifiers, decrypt_chunk_size):
+    print("Trying to decrypt the iOS backup...", end="")
+    backup = EncryptedBackup(
+        backup_directory=base_dir,
+        passphrase=password,
+        cleanup=False,
+        check_same_thread=False,
+        decrypt_chunk_size=decrypt_chunk_size
+    )
+    print("Done\nDecrypting WhatsApp database...", end="")
     try:
         backup.extract_file(
             relative_path=RelativePath.WHATSAPP_MESSAGES,
-            domain=identifiers.DOMAIN,
+            domain_like=identifiers.DOMAIN,
             output_filename=identifiers.MESSAGE
         )
         backup.extract_file(
             relative_path=RelativePath.WHATSAPP_CONTACTS,
-            domain=identifiers.DOMAIN,
+            domain_like=identifiers.DOMAIN,
             output_filename=identifiers.CONTACT
         )
-    except FailedToDecryptError:
+        backup.extract_file(
+            relative_path=RelativePath.WHATSAPP_CALLS,
+            domain_like=identifiers.DOMAIN,
+            output_filename=identifiers.CALL
+        )
+    except ValueError:
         print("Failed to decrypt backup: incorrect password?")
-        exit()
+        exit(7)
+    except FileNotFoundError:
+        print("Essential WhatsApp files are missing from the iOS backup.")
+        exit(6)
     else:
         print("Done")
-    extract_thread = threading.Thread(
-        target=backup.extract_files_by_domain,
-        args=(identifiers.DOMAIN, identifiers.DOMAIN, bplist_reader)
+
+    def extract_progress_handler(file_id, domain, relative_path, n, total_files):
+        if n % 100 == 0:
+            print(f"Decrypting and extracting files...({n}/{total_files})", end="\r")   
+        return True
+
+    backup.extract_files(
+        domain_like=identifiers.DOMAIN,
+        output_folder=identifiers.DOMAIN,
+        preserve_folders=True,
+        filter_callback=extract_progress_handler
     )
-    extract_thread.daemon = True
-    extract_thread.start()
-    dot = 0
-    while extract_thread.is_alive():
-        print(f"Decrypting and extracting files{'.' * dot}{' ' * (3 - dot)}", end="\r")
-        if dot < 3:
-            dot += 1
-            time.sleep(0.5)
-        else:
-            dot = 0
-            time.sleep(0.4)
-    print(f"All required files decrypted and extracted.", end="\n")
-    extract_thread.handled = True
+    print(f"All required files are decrypted and extracted.          ", end="\n")
     return backup
 
 
@@ -70,10 +79,7 @@ def is_encrypted(base_dir):
             return False
 
 
-def extract_media(base_dir, identifiers, preserve_timestamp=False):
-    if preserve_timestamp:
-        from Whatsapp_Chat_Exporter.bplist import BPListReader 
-        preserve_timestamp = BPListReader
+def extract_media(base_dir, identifiers, decrypt_chunk_size):
     if is_encrypted(base_dir):
         if not support_encrypted:
             print("You don't have the dependencies to handle encrypted backup.")
@@ -82,10 +88,11 @@ def extract_media(base_dir, identifiers, preserve_timestamp=False):
             return False
         print("Encryption detected on the backup!")
         password = getpass.getpass("Enter the password for the backup:")
-        extract_encrypted(base_dir, password, identifiers, preserve_timestamp)
+        extract_encrypted(base_dir, password, identifiers, decrypt_chunk_size)
     else:
         wts_db = os.path.join(base_dir, identifiers.MESSAGE[:2], identifiers.MESSAGE)
         contact_db = os.path.join(base_dir, identifiers.CONTACT[:2], identifiers.CONTACT)
+        call_db = os.path.join(base_dir, identifiers.CALL[:2], identifiers.CALL)
         if not os.path.isfile(wts_db):
             if identifiers is WhatsAppIdentifier:
                 print("WhatsApp database not found.")
@@ -95,10 +102,13 @@ def extract_media(base_dir, identifiers, preserve_timestamp=False):
         else:
             shutil.copyfile(wts_db, identifiers.MESSAGE)
         if not os.path.isfile(contact_db):
-            print("Contact database not found.")
-            exit()
+            print("Contact database not found. Skipping...")
         else:
             shutil.copyfile(contact_db, identifiers.CONTACT)
+        if not os.path.isfile(call_db):
+            print("Call database not found. Skipping...")
+        else:
+            shutil.copyfile(call_db, identifiers.CALL)
         _wts_id = identifiers.DOMAIN
         with sqlite3.connect(os.path.join(base_dir, "Manifest.db")) as manifest:
             manifest.row_factory = sqlite3.Row
@@ -136,11 +146,10 @@ def extract_media(base_dir, identifiers, preserve_timestamp=False):
                         pass
                 elif flags == 1:
                     shutil.copyfile(os.path.join(base_dir, folder, hashes), destination)
-                    if preserve_timestamp:
-                        metadata = BPListReader(row["metadata"]).parse()
-                        creation = metadata["$objects"][1]["Birth"]
-                        modification = metadata["$objects"][1]["LastModified"]
-                        os.utime(destination, (modification, modification))
+                    metadata = BPListReader(row["metadata"]).parse()
+                    creation = metadata["$objects"][1]["Birth"]
+                    modification = metadata["$objects"][1]["LastModified"]
+                    os.utime(destination, (modification, modification))
                 if row["_index"] % 100 == 0:
                     print(f"Extracting WhatsApp files...({row['_index']}/{total_row_number})", end="\r")
                 row = c.fetchone()

Whatsapp_Chat_Exporter/utility.py

@@ -1,17 +1,19 @@
 import jinja2
 import json
 import os
-from bleach import clean as sanitize
-from markupsafe import Markup
 import unicodedata
 import re
-from datetime import datetime
+import math
+from bleach import clean as sanitize
+from markupsafe import Markup
+from datetime import datetime, timedelta
 from enum import IntEnum
 from Whatsapp_Chat_Exporter.data_model import ChatStore
 try:
     from enum import StrEnum, IntEnum
 except ImportError:
     # < Python 3.11
+    # This should be removed when the support for Python 3.10 ends.
     from enum import Enum
     class StrEnum(str, Enum):
         pass
@@ -21,6 +23,61 @@ except ImportError:
 
 MAX_SIZE = 4 * 1024 * 1024  # Default 4MB
 ROW_SIZE = 0x3D0
+CURRENT_TZ_OFFSET = datetime.now().astimezone().utcoffset().seconds / 3600
+
+
+def convert_time_unit(time_second: int):
+    time = str(timedelta(seconds=time_second))
+    if "day" not in time:
+        if time_second < 1:
+            time = "less than a second"
+        elif time_second == 1:
+            time = "a second"
+        elif time_second < 60:
+            time = time[5:][1 if time_second < 10 else 0:] + " seconds"
+        elif time_second == 60:
+            time = "a minute"
+        elif time_second < 3600:
+            time = time[2:] + " minutes"
+        elif time_second == 3600:
+            time = "an hour"
+        else:
+            time += " hour"
+    return time
+
+
+def bytes_to_readable(size_bytes: int):
+    """From https://stackoverflow.com/a/14822210/9478891 
+    Authors: james-sapam & other contributors
+    Licensed under CC BY-SA 3.0
+    See git commit logs for changes, if any.
+    """
+    if size_bytes == 0:
+       return "0B"
+    size_name = ("B", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB")
+    i = int(math.floor(math.log(size_bytes, 1024)))
+    p = math.pow(1024, i)
+    s = round(size_bytes / p, 2)
+    return "%s %s" % (s, size_name[i])
+
+
+def readable_to_bytes(size_str: str):
+    SIZE_UNITS = {
+        'B': 1,
+        'KB': 1024,
+        'MB': 1024**2,
+        'GB': 1024**3,
+        'TB': 1024**4,
+        'PB': 1024**5,
+        'EB': 1024**6,
+        'ZB': 1024**7, 
+        'YB': 1024**8
+    }
+    size_str = size_str.upper().strip()
+    number, unit = size_str[:-2].strip(), size_str[-2:].strip()
+    if unit not in SIZE_UNITS or not number.isnumeric():
+        raise ValueError("Invalid input for size_str. Example: 1024GB")
+    return int(number) * SIZE_UNITS[unit]
 
 
 def sanitize_except(html):
@@ -75,13 +132,18 @@ def rendering(
         msgs,
         contact,
         w3css,
-        next,
         chat,
+        headline,
+        next=False,
+        previous=False
     ):
     if chat.their_avatar_thumb is None and chat.their_avatar is not None:
         their_avatar_thumb = chat.their_avatar
     else:
         their_avatar_thumb = chat.their_avatar_thumb
+    if "??" not in headline:
+        raise ValueError("Headline must contain '??' to replace with name")
+    headline = headline.replace("??", name)
     with open(output_file_name, "w", encoding="utf-8") as f:
         f.write(
             template.render(
@@ -92,7 +154,10 @@ def rendering(
                 their_avatar_thumb=their_avatar_thumb,
                 w3css=w3css,
                 next=next,
+                previous=previous,
                 status=chat.status,
+                media_base=chat.media_base,
+                headline=headline
             )
         )
 
@@ -138,11 +203,15 @@ def import_from_json(json_file, data):
         print(f"Importing chats from JSON...({index + 1}/{total_row_number})", end="\r")
 
 
+def sanitize_filename(file_name: str):
+    return "".join(x for x in file_name if x.isalnum() or x in "- ")
+
+
 def get_file_name(contact: str, chat: ChatStore):
     if "@" not in contact and contact not in ("000000000000000", "000000000000001", "ExportedChat"):
         raise ValueError("Unexpected contact format: " + contact)
     phone_number = contact.split('@')[0]
-    if "-" in contact:
+    if "-" in contact and chat.name is not None:
         file_name = ""
     else:
         file_name = phone_number
@@ -150,20 +219,40 @@ def get_file_name(contact: str, chat: ChatStore):
     if chat.name is not None:
         if file_name != "":
             file_name += "-"
-        file_name += chat.name.replace("/", "-")
+        file_name += chat.name.replace("/", "-").replace("\\", "-")
         name = chat.name
     else:
         name = phone_number
 
-    return "".join(x for x in file_name if x.isalnum() or x in "- "), name
+    return sanitize_filename(file_name), name
 
 
-def get_chat_condition(filter, include, column):
+def get_cond_for_empty(enable, jid_field: str, broadcast_field: str):
+    return f"AND (chat.hidden=0 OR {jid_field}='status@broadcast' OR {broadcast_field}>0)" if enable else ""
+
+
+def get_chat_condition(filter, include, columns, jid=None, platform=None):
     if filter is not None:
-        if include:
-            return f'''AND ({' OR '.join(f"{column} LIKE '%{chat}%'" for chat in filter)})'''
-        else:
-            return f'''AND ({' AND '.join(f"{column} NOT LIKE '%{chat}%'" for chat in filter)})'''
+        conditions = []
+        if len(columns) < 2 and jid is not None:
+            raise ValueError("There must be at least two elements in argument columns if jid is not None")
+        if jid is not None:
+            if platform == "android":
+                is_group = f"{jid}.type == 1"
+            elif platform == "ios":
+                is_group = f"{jid} IS NOT NULL"
+            else:
+                raise ValueError("Only android and ios are supported for argument platform if jid is not None")
+        for index, chat in enumerate(filter):
+            if include:
+                conditions.append(f"{' OR' if index > 0 else ''} {columns[0]} LIKE '%{chat}%'")
+                if len(columns) > 1:
+                    conditions.append(f" OR ({columns[1]} LIKE '%{chat}%' AND {is_group})")
+            else:
+                conditions.append(f"{' AND' if index > 0 else ''} {columns[0]} NOT LIKE '%{chat}%'")
+                if len(columns) > 1:
+                    conditions.append(f" AND ({columns[1]} NOT LIKE '%{chat}%' AND {is_group})")
+        return f"AND ({' '.join(conditions)})"
     else:
         return ""
 
@@ -175,6 +264,7 @@ CRYPT14_OFFSETS = (
     {"iv": 66, "db": 99},
     {"iv": 67, "db": 193},
     {"iv": 67, "db": 194},
+    {"iv": 67, "db": 158},
 )
 
 
@@ -291,10 +381,10 @@ def get_status_location(output_folder, offline_static):
     w3css = os.path.join(offline_static, "w3.css")
 
 
-def setup_template(template, no_avatar):
-    if template is None:
+def setup_template(template, no_avatar, experimental=False):
+    if template is None or experimental:
         template_dir = os.path.dirname(__file__)
-        template_file = "whatsapp.html"
+        template_file = "whatsapp.html" if not experimental else template
     else:
         template_dir = os.path.dirname(template)
         template_file = os.path.basename(template)
@@ -308,7 +398,7 @@ def setup_template(template, no_avatar):
     return template_env.get_template(template_file)
 
 # iOS Specific
-APPLE_TIME = datetime.timestamp(datetime(2001, 1, 1))
+APPLE_TIME = 978307200
 
 
 def slugify(value, allow_unicode=False):
@@ -331,6 +421,7 @@ def slugify(value, allow_unicode=False):
 class WhatsAppIdentifier(StrEnum):
     MESSAGE = "7c7fba66680ef796b916b067077cc246adacf01d"
     CONTACT = "b8548dc30aa1030df0ce18ef08b882cf7ab5212f"
+    CALL = "1b432994e958845fffe8e2f190f26d1511534088"
     DOMAIN = "AppDomainGroup-group.net.whatsapp.WhatsApp.shared"
 
 

Whatsapp_Chat_Exporter/vcards_contacts.py

@@ -0,0 +1,82 @@
+import vobject
+from typing import List, TypedDict
+
+
+class ExportedContactNumbers(TypedDict):
+    full_name: str
+    numbers: List[str]
+
+
+class ContactsFromVCards:
+    def __init__(self) -> None:
+        self.contact_mapping = []
+
+    def is_empty(self):
+        return self.contact_mapping == []
+
+    def load_vcf_file(self, vcf_file_path: str, default_country_code: str):
+        self.contact_mapping = read_vcards_file(vcf_file_path, default_country_code)
+
+    def enrich_from_vcards(self, chats):
+        for number, name in self.contact_mapping:
+            # short number must be a bad contact, lets skip it
+            if len(number) <= 5:
+                continue
+
+            for chat in filter_chats_by_prefix(chats, number).values():
+                if not hasattr(chat, 'name') or (hasattr(chat, 'name') and chat.name is None):
+                    setattr(chat, 'name', name)
+
+
+def read_vcards_file(vcf_file_path, default_country_code: str):
+    contacts = []
+    with open(vcf_file_path, mode="r", encoding="utf-8") as f:
+        reader = vobject.readComponents(f)
+        for row in reader:
+            if hasattr(row, 'fn'):
+                name = str(row.fn.value)
+            elif hasattr(row, 'n'):
+                name = str(row.n.value)
+            else:
+                name = None
+            if not hasattr(row, 'tel') or name is None:
+                continue
+            contact: ExportedContactNumbers = {
+                "full_name": name,
+                "numbers": list(map(lambda tel: tel.value, row.tel_list)),
+            }
+            contacts.append(contact)
+
+    return map_number_to_name(contacts, default_country_code)
+
+
+def filter_chats_by_prefix(chats, prefix: str):
+    return {k: v for k, v in chats.items() if k.startswith(prefix)}
+
+
+def map_number_to_name(contacts, default_country_code: str):
+    mapping = []
+    for contact in contacts:
+        for index, num in enumerate(contact['numbers']):
+            normalized = normalize_number(num, default_country_code)
+            if len(contact['numbers']) > 1:
+                name = f"{contact['full_name']} ({index+1})"
+            else:
+                name = contact['full_name']
+            mapping.append((normalized, name))
+    return mapping
+
+
+def normalize_number(number: str, country_code: str):
+    # Clean the number
+    number = ''.join(c for c in number if c.isdigit() or c == "+")
+
+    # A number that starts with a + or 00 means it already have a country code
+    for starting_char in ('+', "00"):
+        if number.startswith(starting_char):
+            return number[len(starting_char):]
+
+    # leading zero should be removed
+    if starting_char == '0':
+        number = number[1:]
+    return country_code + number  # fall back

Whatsapp_Chat_Exporter/vcards_contacts_test.py

@@ -0,0 +1,20 @@
+# from contacts_names_from_vcards import readVCardsFile
+
+from Whatsapp_Chat_Exporter.vcards_contacts import normalize_number, read_vcards_file
+
+
+def test_readVCardsFile():
+    assert len(read_vcards_file("contacts.vcf", "973")) > 0
+
+def test_create_number_to_name_dicts():
+    pass
+
+def test_fuzzy_match_numbers():
+    pass
+
+def test_normalize_number():
+    assert normalize_number('0531234567', '1') == '1531234567'
+    assert normalize_number('001531234567', '2') == '1531234567'
+    assert normalize_number('+1531234567', '34') == '1531234567'
+    assert normalize_number('053(123)4567', '34') == '34531234567'
+    assert normalize_number('0531-234-567', '58') == '58531234567'

Whatsapp_Chat_Exporter/whatsapp.html

@@ -20,7 +20,6 @@
 		}
 		footer {
 			border-top: 2px solid #e3e6e7;
-			font-size: 2em;
 			padding: 20px 0 20px 0;
 		}
 		article {
@@ -87,10 +86,11 @@
 			max-height: 100px !important;
 		}
 		</style>
+	<base href="{{ media_base }}" target="_blank">
 	</head>
 	<body>
 		<header class="w3-center w3-top">
-			Chat history with {{ name }}
+			{{ headline }}
 			{% if status is not none %}
 			<br>
 			<span class="w3-small">{{ status }}</span>
@@ -120,7 +120,7 @@
 							{% if msg.reply is not none %}
 								<div class="reply">
 									<span class="blue">Replying to </span>
-									<a href="#{{msg.reply}}" class="reply_link">
+									<a href="#{{msg.reply}}" target="_self" class="reply_link no-base">
 										{% if msg.quoted_data is not none %}
 											"{{msg.quoted_data}}"
 										{% else %}
@@ -137,21 +137,26 @@
 									<p>{{ msg.data or 'Not supported WhatsApp internal message' }}</p>
 									{% endif %}
 								</div>
+								{% if msg.caption is not none %}
+								<div class="w3-container">
+									{{ msg.caption | urlize(none, true, '_blank') }}
+								</div>
+								{% endif %}
 							{% else %}
 								{% if msg.media == false %}
 								{{ msg.data | sanitize_except() | urlize(none, true, '_blank') }}
 								{% else %}
 									{% if "image/" in msg.mime %}
 									<a href="{{ msg.data }}">
-										<img src="{{ msg.thumb if msg.thumb is not none else msg.data }}" {{ 'class="sticker"' | safe if msg.sticker }} />
+										<img src="{{ msg.thumb if msg.thumb is not none else msg.data }}" {{ 'class="sticker"' | safe if msg.sticker }} loading="lazy"/>
 									</a>
 									{% elif "audio/" in msg.mime %}
 									<audio controls="controls" autobuffer="autobuffer">
 										<source src="{{ msg.data }}" />
 									</audio>
 									{% elif "video/" in msg.mime %}
-									<video controls="controls" autobuffer="autobuffer">
-										<source src="{{ msg.data }}" />
+									<video class="lazy" autobuffer {% if msg.message_type|int == 13 or msg.message_type|int == 11 %}autoplay muted loop playsinline{%else%}controls{% endif %}>
+										<source type="{{ msg.mime }}" data-src="{{ msg.data }}" />
 									</video>
 									{% elif "/" in msg.mime %}
 										<div class="w3-panel w3-border-blue w3-pale-blue w3-rightbar w3-leftbar w3-threequarter w3-center">
@@ -161,17 +166,18 @@
 										{% filter escape %}{{ msg.data }}{% endfilter %}
 									{% endif %}
 									{% if msg.caption is not none %}
-									<br>
-									{{ msg.caption | urlize(none, true, '_blank') }}
+									<div class="w3-container">
+										{{ msg.caption | urlize(none, true, '_blank') }}
+									</div>
 									{% endif %}
 								{% endif %}
-								{% endif %}
+							{% endif %}
 							</div>
 						</div>
 						{% if not no_avatar and my_avatar is not none %}
 						<div class="w3-col m2 l2 pad-left-10">
 							<a href="{{ my_avatar }}">
-								<img src="{{ my_avatar }}" onerror="this.style.display='none'" class="avatar">
+								<img src="{{ my_avatar }}" onerror="this.style.display='none'" class="avatar" loading="lazy">
 							</a>
 						</div>
 						{% endif %}
@@ -191,9 +197,9 @@
 						{% if not no_avatar %}
 						<div class="w3-col m2 l2">
 								{% if their_avatar is not none %}
-								<a href="{{ their_avatar }}"><img src="{{ their_avatar_thumb or '' }}" onerror="this.style.display='none'" class="avatar"></a>
+								<a href="{{ their_avatar }}"><img src="{{ their_avatar_thumb or '' }}" onerror="this.style.display='none'" class="avatar" loading="lazy"></a>
 								{% else %}
-								<img src="{{ their_avatar_thumb or '' }}" onerror="this.style.display='none'" class="avatar">
+								<img src="{{ their_avatar_thumb or '' }}" onerror="this.style.display='none'" class="avatar" loading="lazy">
 								{% endif %}
 						</div>
 						<div class="w3-col m10 l10">
@@ -204,7 +210,7 @@
 							{% if msg.reply is not none %}
 								<div class="reply">
 									<span class="blue">Replying to </span>
-									<a href="#{{msg.reply}}" class="reply_link">
+									<a href="#{{msg.reply}}" target="_self" class="reply_link no-base">
 										{% if msg.quoted_data is not none %}
 											"{{msg.quoted_data}}"
 										{% else %}
@@ -221,21 +227,26 @@
 									<p>{{ msg.data or 'Not supported WhatsApp internal message' }}</p>
 									{% endif %}
 								</div>
+								{% if msg.caption is not none %}
+								<div class="w3-container">
+								{{ msg.caption | urlize(none, true, '_blank') }}
+								</div>
+								{% endif %}
 							{% else %}
 								{% if msg.media == false %}
 								{{ msg.data | sanitize_except() | urlize(none, true, '_blank') }}
 								{% else %}
 									{% if "image/" in msg.mime %}
 									<a href="{{ msg.data }}">
-										<img src="{{ msg.thumb if msg.thumb is not none else msg.data }}" {{ 'class="sticker"' | safe if msg.sticker }} />
+										<img src="{{ msg.thumb if msg.thumb is not none else msg.data }}" {{ 'class="sticker"' | safe if msg.sticker }} loading="lazy"/>
 									</a>
 									{% elif "audio/" in msg.mime %}
 									<audio controls="controls" autobuffer="autobuffer">
 										<source src="{{ msg.data }}" />
 									</audio>
 									{% elif "video/" in msg.mime %}
-									<video controls="controls" autobuffer="autobuffer">
-										<source src="{{ msg.data }}" />
+									<video class="lazy" autobuffer {% if msg.message_type|int == 13 or msg.message_type|int == 11 %}autoplay muted loop playsinline{%else%}controls{% endif %}>
+										<source type="{{ msg.mime }}" data-src="{{ msg.data }}" />
 									</video>
 									{% elif "/" in msg.mime %}
 										<div class="w3-panel w3-border-blue w3-pale-blue w3-rightbar w3-leftbar w3-threequarter w3-center">
@@ -245,8 +256,9 @@
 										{% filter escape %}{{ msg.data }}{% endfilter %}
 									{% endif %}
 									{% if msg.caption is not none %}
-									<br>
-									{{ msg.caption | urlize(none, true, '_blank') }}
+									<div class="w3-container">
+										{{ msg.caption | urlize(none, true, '_blank') }}
+									</div>
 									{% endif %}
 								{% endif %}
 							{% endif %}
@@ -259,11 +271,59 @@
 			</div>
 		</article>
 		<footer class="w3-center">
-			{% if next %}
-			<a href="./{{ next }}">Next</a>
-			{% else %}
-			End of history
+			<h2>
+			{% if previous %}
+			<a href="./{{ previous }}" target="_self">Previous</a>
 			{% endif %}
+			<h2>
+			{% if next %}
+			<a href="./{{ next }}" target="_self">Next</a>
+			{% else %}
+			End of History
+			{% endif %}
+			</h2>
+			<br>
+			Portions of this page are reproduced from <a href="https://web.dev/articles/lazy-loading-video">work</a> created and <a href="https://developers.google.com/readme/policies">shared by Google</a> and used according to terms described in the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache 2.0 License</a>. 
 		</footer>
+		<script>
+		document.addEventListener("DOMContentLoaded", function() {
+		var lazyVideos = [].slice.call(document.querySelectorAll("video.lazy"));
+
+		if ("IntersectionObserver" in window) {
+			var lazyVideoObserver = new IntersectionObserver(function(entries, observer) {
+			entries.forEach(function(video) {
+				if (video.isIntersecting) {
+				for (var source in video.target.children) {
+					var videoSource = video.target.children[source];
+					if (typeof videoSource.tagName === "string" && videoSource.tagName === "SOURCE") {
+					videoSource.src = videoSource.dataset.src;
+					}
+				}
+
+				video.target.load();
+				video.target.classList.remove("lazy");
+				lazyVideoObserver.unobserve(video.target);
+				}
+			});
+			});
+
+			lazyVideos.forEach(function(lazyVideo) {
+			lazyVideoObserver.observe(lazyVideo);
+			});
+		}
+		});
+		</script>
+		<script>   
+		// Prevent the <base> tag from affecting links with the class "no-base"
+		document.querySelectorAll('.no-base').forEach(link => {
+			link.addEventListener('click', function(event) {
+			const href = this.getAttribute('href');
+			if (href.startsWith('#')) {
+				window.location.hash = href;
+				event.preventDefault();
+			}
+			});
+		});
+		</script>
 	</body>
 </html>

Whatsapp_Chat_Exporter/whatsapp_new.html

@@ -0,0 +1,424 @@
+<!DOCTYPE html>
+<html>
+    <head>
+        <title>Whatsapp - {{ name }}</title> 
+        <meta charset="UTF-8">
+        <script src="https://cdn.tailwindcss.com"></script>
+        <script>
+            tailwind.config = {
+                theme: {
+                    extend: {
+                        colors: {
+                            whatsapp: {
+                                light: '#e7ffdb',
+                                DEFAULT: '#25D366',
+                                dark: '#075E54',
+                                chat: '#efeae2',
+                                'chat-light': '#f0f2f5',
+                            }
+                        }
+                    }
+                }
+            }
+        </script>
+        <style>
+        body, html {
+            height: 100%;
+            margin: 0;
+            padding: 0;
+            scroll-behavior: smooth !important;
+        }
+        .chat-list {
+            height: calc(100vh - 120px);
+            overflow-y: auto;
+        }
+        .message-list {
+            height: calc(100vh - 90px);
+            overflow-y: auto;
+        }
+        @media (max-width: 640px) {
+            .chat-list, .message-list {
+                height: calc(100vh - 108px);
+            }
+        }
+        header {
+            position: fixed;
+            z-index: 20;
+            border-bottom: 2px solid #e3e6e7;
+            font-size: 2em;
+            font-weight: bolder;
+            background-color: white;
+            padding: 20px 0 20px 0;
+        }
+        footer {
+            margin-top: 10px;
+            border-top: 2px solid #e3e6e7;
+            padding: 20px 0 20px 0;
+        }
+        article {
+            width:430px;
+            margin: auto;
+            z-index:10;
+            font-size: 15px;
+            word-wrap: break-word;
+        }
+        img, video, audio{
+            max-width:100%;
+            box-sizing: border-box;
+        }
+        div.reply{
+            font-size: 13px;
+            text-decoration: none;
+        }
+        div:target::before {
+            content: '';
+            display: block;
+            height: 115px;
+            margin-top: -115px;
+            visibility: hidden;
+        }
+        div:target {
+            animation: 3s highlight;
+        }
+        .avatar {
+            border-radius:50%;
+            overflow:hidden;
+            max-width: 64px;
+            max-height: 64px;
+        }
+        .name {
+            color: #3892da;
+        }
+        .pad-left-10 {
+            padding-left: 10px;
+        }
+        .pad-right-10 {
+            padding-right: 10px;
+        }
+        .reply_link {
+            color: #168acc;
+        }
+        .blue {
+            color: #70777a;
+        }
+        .sticker {
+            max-width: 100px !important;
+            max-height: 100px !important;
+        }
+        @keyframes highlight {
+            from {
+                background-color: rgba(37, 211, 102, 0.1);
+            }
+            to {
+                background-color: transparent;
+            }
+        }
+        .search-input {
+            transform: translateY(-100%);
+            transition: transform 0.3s ease-in-out;
+        }
+        .search-input.active {
+            transform: translateY(0);
+        }
+        .reply-box:active {
+            background-color:rgb(200 202 205 / var(--tw-bg-opacity, 1));
+        }
+        </style>
+        <script>
+            function search(event) {
+                keywords = document.getElementById("mainHeaderSearchInput").value;
+                hits = [];
+                document.querySelectorAll(".message-text").forEach(elem => {
+                    if (elem.innerText.trim().includes(keywords)){
+                        hits.push(elem.parentElement.parentElement.id);
+                    }
+                })
+                console.log(hits);
+            }
+        </script>
+    <base href="{{ media_base }}" target="_blank">
+    </head>
+    <body>
+        <article class="h-screen bg-whatsapp-chat-light">
+            <div class="w-full flex flex-col">
+                <div class="p-3 bg-whatsapp-dark flex items-center justify-between border-l border-[#d1d7db]">
+                    <div class="flex items-center">
+                        {% if not no_avatar %}
+                        <div class="w3-col m2 l2">
+                            {% if their_avatar is not none %}
+                            <a href="{{ their_avatar }}"><img src="{{ their_avatar_thumb or '' }}" onerror="this.style.display='none'" class="w-10 h-10 rounded-full mr-3" loading="lazy"></a>
+                            {% else %}
+                            <img src="{{ their_avatar_thumb or '' }}" onerror="this.style.display='none'" class="w-10 h-10 rounded-full mr-3" loading="lazy">
+                            {% endif %}
+                        </div>
+                        {% endif %}
+                        <div>
+                            <h2 class="text-white font-medium">{{ headline }}</h2>
+                            {% if status is not none %}<p class="text-[#8696a0] text-xs">{{ status }}</p>{% endif %}
+                        </div>
+                    </div>
+                    <div class="flex space-x-4">
+                        <!-- <button id="searchButton">
+                            <svg xmlns="http://www.w3.org/2000/svg" class="h-5 w-5 text-[#aebac1]" fill="none" viewBox="0 0 24 24" stroke="currentColor">
+                                <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M21 21l-6-6m2-5a7 7 0 11-14 0 7 7 0 0114 0z" />
+                            </svg>
+                        </button> -->
+                        <!-- <svg xmlns="http://www.w3.org/2000/svg" class="h-5 w-5 text-[#aebac1]" fill="none" viewBox="0 0 24 24" stroke="currentColor">
+                            <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 19l-7-7 7-7" />
+                        </svg> -->
+                        {% if previous %}
+                        <a href="./{{ previous }}" target="_self">
+                            <svg xmlns="http://www.w3.org/2000/svg" class="h-5 w-5 text-[#aebac1]" fill="none" viewBox="0 0 24 24" stroke="currentColor">
+                                <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 5l-7 7 7 7" />
+                            </svg>
+                        </a>
+                        {% endif %}
+                        {% if next %}
+                        <a href="./{{ next }}" target="_self">
+                            <svg xmlns="http://www.w3.org/2000/svg" class="h-5 w-5 text-[#aebac1]" fill="none" viewBox="0 0 24 24" stroke="currentColor">
+                                <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 5l7 7-7 7" />
+                            </svg>
+                        </a>
+                        {% endif %}
+                    </div>
+                    <!-- Search Input Overlay -->
+                    <div id="mainSearchInput" class="search-input absolute article top-0 bg-whatsapp-dark p-3 flex items-center space-x-3">
+                        <button id="closeMainSearch" class="text-[#aebac1]">
+                            <svg xmlns="http://www.w3.org/2000/svg" class="h-6 w-6" fill="none" viewBox="0 0 24 24" stroke="currentColor">
+                                <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 19l-7-7 7-7" />
+                            </svg>
+                        </button>
+                        <input type="text" placeholder="Search..." class="flex-1 bg-[#1f2c34] text-white rounded-lg px-3 py-1 focus:outline-none" id="mainHeaderSearchInput" onkeyup="search(event)">
+                    </div>
+                </div>
+            </div>
+            <div class="flex-1 p-5 message-list">
+                <div class="flex flex-col space-y-2">
+                    <!--Date-->
+                    {% set last = {'last': 946688461.001} %}
+                    {% for msg in msgs -%}
+                        {% if determine_day(last.last, msg.timestamp) is not none  %}
+                            <div class="flex justify-center">
+                                <div class="bg-[#e1f2fb] rounded-lg px-2 py-1 text-xs text-[#54656f]">
+                                    {{ determine_day(last.last, msg.timestamp) }}
+                                </div>
+                            </div>
+                            {% if last.update({'last': msg.timestamp}) %}{% endif %}
+                        {% endif %}
+                        <!--Actual messages-->
+                        {% if msg.from_me == true %}
+                        <div class="flex justify-end" id="{{ msg.key_id }}">
+                            <div class="bg-whatsapp-light rounded-lg p-2 max-w-[80%] shadow-sm">
+                                {% if msg.reply is not none %}
+                                <a href="#{{msg.reply}}" target="_self" class="no-base">
+                                    <div class="mb-2 p-1 bg-whatsapp-chat-light rounded border-l-4 border-whatsapp text-sm reply-box">
+                                        <p class="text-whatsapp font-medium text-xs">Replying to</p>
+                                        <p class="text-[#111b21] text-xs truncate">
+                                            {% if msg.quoted_data is not none %}
+                                                "{{msg.quoted_data}}"
+                                            {% else %}
+                                                this message
+                                            {% endif %}
+                                        </p>
+                                    </div>
+                                </a>
+                                {% endif %}
+                                <p class="text-[#111b21] text-sm message-text">
+                                    {% if msg.meta == true or msg.media == false and msg.data is none %}     
+                                        <div class="flex justify-center mb-2">
+                                            <div class="bg-[#FFF3C5] rounded-lg px-3 py-2 text-sm text-[#856404] flex items-center">
+                                                {% if msg.safe %}
+                                                    {{ msg.data | safe or 'Not supported WhatsApp internal message' }}
+                                                {% else %}
+                                                    {{ msg.data or 'Not supported WhatsApp internal message' }}
+                                                {% endif %}
+                                            </div>
+                                        </div>
+                                        {% if msg.caption is not none %}
+                                            <p>{{ msg.caption | urlize(none, true, '_blank') }}</p>
+                                        {% endif %}
+                                    {% else %}
+                                        {% if msg.media == false %}
+                                            {{ msg.data | sanitize_except() | urlize(none, true, '_blank') }}
+                                        {% else %}
+                                            {% if "image/" in msg.mime %}
+                                                <a href="{{ msg.data }}">
+                                                    <img src="{{ msg.thumb if msg.thumb is not none else msg.data }}" {{ 'class="sticker"' | safe if msg.sticker }} loading="lazy"/>
+                                                </a>
+                                            {% elif "audio/" in msg.mime %}
+                                                <audio controls="controls" autobuffer="autobuffer">
+                                                    <source src="{{ msg.data }}" />
+                                                </audio>
+                                            {% elif "video/" in msg.mime %}
+                                                <video class="lazy" autobuffer {% if msg.message_type|int == 13 or msg.message_type|int == 11 %}autoplay muted loop playsinline{%else%}controls{% endif %}>
+                                                    <source type="{{ msg.mime }}" data-src="{{ msg.data }}" />
+                                                </video>
+                                            {% elif "/" in msg.mime %}
+                                                The file cannot be displayed here, however it should be located at <a href="./{{ msg.data }}">here</a>
+                                            {% else %}
+                                                {% filter escape %}{{ msg.data }}{% endfilter %}
+                                            {% endif %}
+                                            {% if msg.caption is not none %}
+                                                {{ msg.caption | urlize(none, true, '_blank') }}
+                                            {% endif %}
+                                        {% endif %}
+                                    {% endif %}
+                                </p>
+                                <p class="text-[10px] text-[#667781] text-right mt-1">{{ msg.time }}</p>
+                            </div>
+                        </div>
+                        {% else %}
+                        <div class="flex justify-start" id="{{ msg.key_id }}">
+                            <div class="bg-white rounded-lg p-2 max-w-[80%] shadow-sm">
+                                {% if msg.reply is not none %}
+                                <a href="#{{msg.reply}}" target="_self" class="no-base">
+                                    <div class="mb-2 p-1 bg-whatsapp-chat-light rounded border-l-4 border-whatsapp text-sm reply-box">
+                                        <p class="text-whatsapp font-medium text-xs">Replying to</p>
+                                        <p class="text-[#808080] text-xs truncate">
+                                            {% if msg.quoted_data is not none %}
+                                                {{msg.quoted_data}}
+                                            {% else %}
+                                                this message
+                                            {% endif %}
+                                        </p>
+                                    </div>
+                                </a>
+                                {% endif %}
+                                <p class="text-[#111b21] text-sm">
+                                    {% if msg.meta == true or msg.media == false and msg.data is none %}
+                                        <div class="flex justify-center mb-2">
+                                            <div class="bg-[#FFF3C5] rounded-lg px-3 py-2 text-sm text-[#856404] flex items-center">
+                                                {% if msg.safe %}
+                                                    {{ msg.data | safe or 'Not supported WhatsApp internal message' }}
+                                                {% else %}
+                                                    {{ msg.data or 'Not supported WhatsApp internal message' }}
+                                                {% endif %}
+                                            </div>
+                                        </div>
+                                        {% if msg.caption is not none %}
+                                            <p>{{ msg.caption | urlize(none, true, '_blank') }}</p>
+                                        {% endif %}
+                                    {% else %}
+                                        {% if msg.media == false %}
+                                            {{ msg.data | sanitize_except() | urlize(none, true, '_blank') }}
+                                        {% else %}
+                                            {% if "image/" in msg.mime %}
+                                                <a href="{{ msg.data }}">
+                                                    <img src="{{ msg.thumb if msg.thumb is not none else msg.data }}" {{ 'class="sticker"' | safe if msg.sticker }} loading="lazy"/>
+                                                </a>
+                                            {% elif "audio/" in msg.mime %}
+                                                <audio controls="controls" autobuffer="autobuffer">
+                                                    <source src="{{ msg.data }}" />
+                                                </audio>
+                                            {% elif "video/" in msg.mime %}
+                                                <video class="lazy" autobuffer {% if msg.message_type|int == 13 or msg.message_type|int == 11 %}autoplay muted loop playsinline{%else%}controls{% endif %}>
+                                                    <source type="{{ msg.mime }}" data-src="{{ msg.data }}" />
+                                                </video>
+                                            {% elif "/" in msg.mime %}
+                                                The file cannot be displayed here, however it should be located at <a href="./{{ msg.data }}">here</a>
+                                            {% else %}
+                                                {% filter escape %}{{ msg.data }}{% endfilter %}
+                                            {% endif %}
+                                            {% if msg.caption is not none %}
+                                                {{ msg.caption | urlize(none, true, '_blank') }}
+                                            {% endif %}
+                                        {% endif %}
+                                    {% endif %}
+                                </p>
+                                <div class="flex items-baseline text-[10px] text-[#667781] mt-1 gap-2">
+                                    <span class="flex-shrink-0">
+                                        {% if msg.sender is not none %}
+                                        {{ msg.sender }}
+                                        {% endif %}
+                                    </span>
+                                    <span class="flex-grow min-w-[4px]"></span>
+                                    <span class="flex-shrink-0">{{ msg.time }}</span>
+                                    </div>
+                            </div>
+                        </div>
+                        {% endif %}
+                    {% endfor %}
+                </div>
+                <footer>
+                    <h2 class="text-center">
+                    {% if not next %}
+                    End of History
+                    {% endif %}
+                    </h2>
+                    <br>
+                    Portions of this page are reproduced from <a href="https://web.dev/articles/lazy-loading-video">work</a> created and <a href="https://developers.google.com/readme/policies">shared by Google</a> and used according to terms described in the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache 2.0 License</a>. 
+                </footer>
+            </div>
+        </article>
+    </body>
+    <script>
+    // Search functionality
+    const searchButton = document.getElementById('searchButton');
+    const mainSearchInput = document.getElementById('mainSearchInput');
+    const closeMainSearch = document.getElementById('closeMainSearch');
+    const mainHeaderSearchInput = document.getElementById('mainHeaderSearchInput');
+
+    // Function to show search input
+    const showSearch = () => {
+        mainSearchInput.classList.add('active');
+        mainHeaderSearchInput.focus();
+    };
+
+    // Function to hide search input
+    const hideSearch = () => {
+        mainSearchInput.classList.remove('active');
+        mainHeaderSearchInput.value = '';
+    };
+
+    // Event listeners
+    searchButton.addEventListener('click', showSearch);
+    closeMainSearch.addEventListener('click', hideSearch);
+
+    // Handle ESC key
+    document.addEventListener('keydown', (event) => {
+        if (event.key === 'Escape' && mainSearchInput.classList.contains('active')) {
+            hideSearch();
+        }
+    });
+    </script>
+    <script>
+    document.addEventListener("DOMContentLoaded", function() {
+    var lazyVideos = [].slice.call(document.querySelectorAll("video.lazy"));
+
+    if ("IntersectionObserver" in window) {
+        var lazyVideoObserver = new IntersectionObserver(function(entries, observer) {
+        entries.forEach(function(video) {
+            if (video.isIntersecting) {
+            for (var source in video.target.children) {
+                var videoSource = video.target.children[source];
+                if (typeof videoSource.tagName === "string" && videoSource.tagName === "SOURCE") {
+                videoSource.src = videoSource.dataset.src;
+                }
+            }
+
+            video.target.load();
+            video.target.classList.remove("lazy");
+            lazyVideoObserver.unobserve(video.target);
+            }
+        });
+        });
+
+        lazyVideos.forEach(function(lazyVideo) {
+        lazyVideoObserver.observe(lazyVideo);
+        });
+    }
+    });
+    </script>
+    <script>   
+    // Prevent the <base> tag from affecting links with the class "no-base"
+    document.querySelectorAll('.no-base').forEach(link => {
+        link.addEventListener('click', function(event) {
+        const href = this.getAttribute('href');
+        if (href.startsWith('#')) {
+            window.location.hash = href;
+            event.preventDefault();
+        }
+        });
+    });
+    </script>
+</html>

docs.html

@@ -1,7 +1,6 @@
 <!DOCTYPE html>
 <html>
     <head>
-        <meta http-equiv="refresh" content="0; url='https://github.com/KnugiHK/WhatsApp-Chat-Exporter/wiki'" />
         <script type="text/javascript">
             destination = {
                 "filter": "Filter",

pyproject.toml

@@ -0,0 +1,61 @@
+[build-system]
+requires = ["setuptools", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "whatsapp-chat-exporter"
+version = "0.11.0"
+description = "A Whatsapp database parser that provides history of your Whatsapp conversations in HTML and JSON. Android, iOS, iPadOS, Crypt12, Crypt14, Crypt15 supported."
+readme = "README.md"
+authors = [
+    { name = "KnugiHK", email = "hello@knugi.com" }
+]
+license = { text = "MIT" }
+keywords = [
+    "android", "ios", "parsing", "history", "iphone", "message", "crypt15",
+    "customizable", "whatsapp", "android-backup", "messages", "crypt14", 
+    "crypt12", "whatsapp-chat-exporter", "whatsapp-export", "iphone-backup",
+    "whatsapp-database", "whatsapp-database-parser", "whatsapp-conversations"
+]
+classifiers = [
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Intended Audience :: End Users/Desktop",
+    "Topic :: Communications :: Chat",
+    "Topic :: Utilities",
+    "Topic :: Database"
+]
+requires-python = ">=3.9"
+dependencies = [
+    "jinja2",
+    "bleach"
+]
+
+[project.optional-dependencies]
+android_backup = ["pycryptodome", "javaobj-py3"]
+crypt12 = ["pycryptodome"]
+crypt14 = ["pycryptodome"]
+crypt15 = ["pycryptodome", "javaobj-py3"]
+all = ["pycryptodome", "javaobj-py3", "vobject"]
+everything = ["pycryptodome", "javaobj-py3", "vobject"]
+backup = ["pycryptodome", "javaobj-py3"]
+vcards = ["vobject", "pycryptodome", "javaobj-py3"]
+
+[project.scripts]
+wtsexporter = "Whatsapp_Chat_Exporter.__main__:main"
+waexporter = "Whatsapp_Chat_Exporter.__main__:main"
+whatsapp-chat-exporter = "Whatsapp_Chat_Exporter.__main__:main"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["Whatsapp_Chat_Exporter"]
+
+[tool.setuptools.package-data]
+template = ["whatsapp.html"]

setup.py

@@ -1,69 +0,0 @@
-import setuptools
-from re import search
-
-with open("README.md", "r") as fh:
-    long_description = fh.read()
-
-with open("Whatsapp_Chat_Exporter/__init__.py", encoding="utf8") as f:
-    version = search(r'__version__ = "(.*?)"', f.read()).group(1)
-
-setuptools.setup(
-    name="whatsapp-chat-exporter",
-    version=version,
-    author="KnugiHK",
-    author_email="hello@knugi.com",
-    description=("A Whatsapp database parser that will give you the "
-                "history of your Whatsapp conversations in HTML and JSON. "
-                "Android, iOS, iPadOS, Crypt12, Crypt14, Crypt15 supported."),
-    long_description=long_description,
-    long_description_content_type="text/markdown",
-    license="MIT",
-    keywords=[
-        "android", "ios", "parsing", "history", "iphone", "message", "crypt15",
-        "customizable", "whatsapp", "android-backup", "messages", "crypt14", 
-        "crypt12", "whatsapp-chat-exporter", "whatsapp-export", "iphone-backup",
-        "whatsapp-database", "whatsapp-database-parser", "whatsapp-conversations"
-    ],
-    platforms=["any"],
-    url="https://github.com/KnugiHK/Whatsapp-Chat-Exporter",
-    packages=setuptools.find_packages(),
-    package_data={
-        '': ['whatsapp.html']
-    },
-    classifiers=[
-        "Programming Language :: Python :: 3 :: Only",
-        "Programming Language :: Python :: 3.8",
-        "Programming Language :: Python :: 3.9",
-        "Programming Language :: Python :: 3.10",
-        "Programming Language :: Python :: 3.11",
-        "License :: OSI Approved :: MIT License",
-        "Operating System :: OS Independent",
-        "Development Status :: 4 - Beta",
-        "Environment :: Console",
-        "Intended Audience :: End Users/Desktop",
-        "Topic :: Communications :: Chat",
-        "Topic :: Utilities",
-        "Topic :: Database"
-    ],
-    python_requires='>=3.8',
-    install_requires=[
-        'jinja2',
-        'bleach'
-    ],
-    extras_require={
-        'android_backup':  ["pycryptodome", "javaobj-py3"],
-        'crypt12':  ["pycryptodome"],
-        'crypt14':  ["pycryptodome"],
-        'crypt15':  ["pycryptodome", "javaobj-py3"],
-        'all': ["pycryptodome", "javaobj-py3"],
-        'everything': ["pycryptodome", "javaobj-py3"],
-        'backup': ["pycryptodome", "javaobj-py3"]
-    },
-    entry_points={
-        "console_scripts": [
-            "wtsexporter = Whatsapp_Chat_Exporter.__main__:main",
-            "waexporter = Whatsapp_Chat_Exporter.__main__:main",
-            "whatsapp-chat-exporter = Whatsapp_Chat_Exporter.__main__:main"
-        ]
-    }
-)