Revert back to 3.10 as 3.11 and 3.12 failed

Add support for separating media files per chat

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,35 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[BUG]"
+labels: ''
+assignees: ''
+
+---
+
+# Must have
+- WhatsApp version: [WhatsApp version] - [Android/iOS]
+- Platform: [Linux/Windows/MacOS]
+- Branch and version: [main/dev] - [exporter version]
+
+If it is an error yield by Python, please also provide the trackback
+```
+[trackback here]
+```
+
+# Nice to have
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,17 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: "[FEATURE]"
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/workflows/compile-binary.yml

@@ -16,7 +16,7 @@ jobs:
     - name: Set up Python
       uses: actions/setup-python@v4
       with:
-        python-version: '3.10'
+        python-version: '3.11'
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
@@ -24,7 +24,7 @@ jobs:
         pip install .
     - name: Build binary with Nuitka
       run: |
-        python -m nuitka --onefile --include-data-file=./Whatsapp_Chat_Exporter/whatsapp.html=./Whatsapp_Chat_Exporter/whatsapp.html --follow-imports Whatsapp_Chat_Exporter/__main__.py 
+        python -m nuitka --no-deployment-flag=self-execution --onefile --include-data-file=./Whatsapp_Chat_Exporter/whatsapp.html=./Whatsapp_Chat_Exporter/whatsapp.html --follow-imports Whatsapp_Chat_Exporter/__main__.py 
         cp __main__.bin wtsexporter_linux_x64
         sha256sum wtsexporter_linux_x64
     - uses: actions/upload-artifact@v3
@@ -40,7 +40,7 @@ jobs:
     - name: Set up Python
       uses: actions/setup-python@v4
       with:
-        python-version: '3.10'
+        python-version: '3.11'
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
@@ -48,7 +48,7 @@ jobs:
         pip install .
     - name: Build binary with Nuitka
       run: |
-        python -m nuitka --onefile --include-data-file=./Whatsapp_Chat_Exporter/whatsapp.html=./Whatsapp_Chat_Exporter/whatsapp.html --assume-yes-for-downloads --follow-imports Whatsapp_Chat_Exporter\__main__.py 
+        python -m nuitka --no-deployment-flag=self-execution --onefile --include-data-file=./Whatsapp_Chat_Exporter/whatsapp.html=./Whatsapp_Chat_Exporter/whatsapp.html --assume-yes-for-downloads --follow-imports Whatsapp_Chat_Exporter\__main__.py 
         copy __main__.exe wtsexporter_x64.exe
         Get-FileHash wtsexporter_x64.exe
     - uses: actions/upload-artifact@v3
@@ -72,7 +72,7 @@ jobs:
         pip install .
     - name: Build binary with Nuitka
       run: |
-        python -m nuitka --onefile --include-data-file=./Whatsapp_Chat_Exporter/whatsapp.html=./Whatsapp_Chat_Exporter/whatsapp.html --follow-imports Whatsapp_Chat_Exporter/__main__.py 
+        python -m nuitka --no-deployment-flag=self-execution --onefile --include-data-file=./Whatsapp_Chat_Exporter/whatsapp.html=./Whatsapp_Chat_Exporter/whatsapp.html --follow-imports Whatsapp_Chat_Exporter/__main__.py 
         cp __main__.bin wtsexporter_macos_x64
         shasum -a 256 wtsexporter_macos_x64
     - uses: actions/upload-artifact@v3

.github/workflows/python-publish.yml

@@ -18,9 +18,9 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
     - name: Set up Python
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v5
       with:
         python-version: '3.x'
     - name: Install dependencies
@@ -31,6 +31,3 @@ jobs:
       run: python -m build
     - name: Publish package
       uses: pypa/gh-action-pypi-publish@release/v1
-      with:
-        user: __token__
-        password: ${{ secrets.PYPI_API_TOKEN }}

.gitignore

@@ -127,3 +127,10 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# Nuitka
+*.build/
+*.dist/
+*.onefile-build/
+*.exe
+__main__

CNAME

@@ -1 +1 @@
-wts.knugi.com
+wts.knugi.dev

LICENSE.django

@@ -0,0 +1,36 @@
+The Whatsapp Chat Exporter is licensed under the MIT license. For more information,
+refer to the file LICENSE.
+
+Whatsapp Chat Exporter incorporates code from Django, governed by the three-clause
+BSD license—a permissive open-source license. The copyright and license details are
+provided below to adhere to Django's terms.
+
+------
+
+Copyright (c) Django Software Foundation and individual contributors.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+    1. Redistributions of source code must retain the above copyright notice,
+       this list of conditions and the following disclaimer.
+
+    2. Redistributions in binary form must reproduce the above copyright
+       notice, this list of conditions and the following disclaimer in the
+       documentation and/or other materials provided with the distribution.
+
+    3. Neither the name of Django nor the names of its contributors may be used
+       to endorse or promote products derived from this software without
+       specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -99,7 +99,7 @@ wtsexporter -i -b "C:\Users\[Username]\AppData\Roaming\Apple Computer\MobileSync
 ```
 #### Mac
 ```sh
-wtsexporter -i -b "~/Library/Application Support/MobileSync/Backup/[device id]"
+wtsexporter -i -b ~/Library/Application\ Support/MobileSync/Backup/[device id]
 ```
 
 ## Results
@@ -110,19 +110,23 @@ After extracting, you will get these:
 #### Group Message
 ![Group Message](imgs/group.png)
 
-*The above screenshots were taken longgggggggggggggg ago. I am going to update them when possible.*
-
 ## More options
 Invoke the wtsexporter with --help option will show you all options available.
 ```sh
 > wtsexporter --help
-usage: wtsexporter [-h] [-a] [-i] [-e EXPORTED] [-w WA] [-m MEDIA] [-b BACKUP] [-o OUTPUT] [-j [JSON]] [-d DB] [-k KEY] [-t TEMPLATE] [-s] [-c] [--offline OFFLINE] [--size [SIZE]]
-                   [--no-html] [--check-update] [--assume-first-as-me]
+usage: wtsexporter [-h] [-a] [-i] [-e EXPORTED] [-w WA] [-m MEDIA] [-b BACKUP] [-o OUTPUT] [-j [JSON]] [-d DB]
+                   [-k KEY] [-t TEMPLATE] [-s] [-c] [--offline OFFLINE] [--size [SIZE]] [--no-html] [--check-update]
+                   [--assume-first-as-me] [--no-avatar] [--import] [--business] [--preserve-timestamp] [--wab WAB]
+                   [--time-offset {-12 to 14}] [--date DATE] [--date-format FORMAT] [--include [phone number ...]]
+                   [--exclude [phone number ...]] [--create-separated-media]
+
+A customizable Android and iPhone WhatsApp database parser that will give you the history of your WhatsApp
+conversations in HTML and JSON. Android Backup Crypt12, Crypt14 and Crypt15 supported.
 
 options:
   -h, --help            show this help message and exit
   -a, --android         Define the target as Android
-  -i, --iphone, --ios   Define the target as iPhone/iPad
+  -i, --ios, --iphone   Define the target as iPhone/iPad
   -e EXPORTED, --exported EXPORTED
                         Define the target as exported chat file and specify the path to the file
   -w WA, --wa WA        Path to contact database (default: wa.db/ContactsV2.sqlite)
@@ -146,8 +150,25 @@ options:
   --no-html             Do not output html files
   --check-update        Check for updates (require Internet access)
   --assume-first-as-me  Assume the first message in a chat as sent by me (must be used together with -e)
+  --no-avatar           Do not render avatar in HTML output
+  --import              Import JSON file and convert to HTML output
+  --business            Use Whatsapp Business default files (iOS only)
+  --preserve-timestamp  Preserve the modification timestamp of the extracted files (iOS only)
+  --wab WAB, --wa-backup WAB
+                        Path to contact database in crypt15 format
+  --time-offset {-12 to 14}
+                        Offset in hours (-12 to 14) for time displayed in the output
+  --date DATE           The date filter in specific format (inclusive)
+  --date-format FORMAT  The date format for the date filter
+  --include [phone number ...]
+                        Include chats that match the supplied phone number
+  --exclude [phone number ...]
+                        Exclude chats that match the supplied phone number
+  --create-separated-media
+                        Create a copy of the media seperated per chat in <MEDIA>/separated/ directory 
+                        (Android only)
 
-WhatsApp Chat Exporter: 0.9.5 Licensed with MIT
+WhatsApp Chat Exporter: 0.10.0 Licensed with MIT
 ```
 
 # To do
@@ -156,6 +177,8 @@ See [issues](https://github.com/KnugiHK/Whatsapp-Chat-Exporter/issues).
 # Copyright
 This is a MIT licensed project.
 
-The Telegram Desktop's export is the reference for whatsapp.html in this repo
+The Telegram Desktop's export is the reference for whatsapp.html in this repo.
+
+bplist.py was released by Vladimir "Farcaller" Pouzanov under MIT license.
 
 WhatsApp Chat Exporter is not affiliated, associated, authorized, endorsed by, or in any way officially connected with the WhatsApp LLC, or any of its subsidiaries or its affiliates. The official WhatsApp LLC website can be found at https://www.whatsapp.com/.

Whatsapp_Chat_Exporter/__init__.py

@@ -1,3 +1,3 @@
 #!/usr/bin/python3
 
-__version__ = "0.9.5"
+__version__ = "0.10.0"

Whatsapp_Chat_Exporter/__main__.py

@@ -1,16 +1,19 @@
 #!/usr/bin/python3
 
+import io
 import os
 import sqlite3
 import shutil
 import json
 import string
 import glob
-from Whatsapp_Chat_Exporter import extract_exported, extract_iphone
-from Whatsapp_Chat_Exporter import extract, extract_iphone_media
+from Whatsapp_Chat_Exporter import exported_handler, android_handler
+from Whatsapp_Chat_Exporter import ios_handler, ios_media_handler
 from Whatsapp_Chat_Exporter.data_model import ChatStore
-from Whatsapp_Chat_Exporter.utility import Crypt, check_update, import_from_json
+from Whatsapp_Chat_Exporter.utility import APPLE_TIME, Crypt, DbType
+from Whatsapp_Chat_Exporter.utility import check_update, import_from_json
 from argparse import ArgumentParser, SUPPRESS
+from datetime import datetime
 from sys import exit
 try:
     from .__init__ import __version__
@@ -20,10 +23,11 @@ except ImportError:
 
 def main():
     parser = ArgumentParser(
-        description = 'A customizable Android and iPhone WhatsApp database parser that '
-                      'will give you the history of your WhatsApp conversations inHTML '
+        description = 'A customizable Android and iOS/iPadOS WhatsApp database parser that '
+                      'will give you the history of your WhatsApp conversations in HTML '
                       'and JSON. Android Backup Crypt12, Crypt14 and Crypt15 supported.',
-        epilog = f'WhatsApp Chat Exporter: {__version__} Licensed with MIT'
+        epilog = f'WhatsApp Chat Exporter: {__version__} Licensed with MIT. See'
+                  'https://wts.knugi.dev/docs?dest=osl for all open source licenses.'
     )
     parser.add_argument(
         '-a',
@@ -34,9 +38,9 @@ def main():
         help="Define the target as Android")
     parser.add_argument(
         '-i',
-        '--iphone',
         '--ios',
-        dest='iphone',
+        '--iphone',
+        dest='ios',
         default=False,
         action='store_true',
         help="Define the target as iPhone/iPad")
@@ -65,7 +69,7 @@ def main():
         dest="backup",
         default=None,
         help="Path to Android (must be used together "
-             "with -k)/iPhone WhatsApp backup")
+             "with -k)/iOS WhatsApp backup")
     parser.add_argument(
         "-o",
         "--output",
@@ -177,6 +181,78 @@ def main():
         action='store_true',
         help="Import JSON file and convert to HTML output"
     )
+    parser.add_argument(
+        "--business",
+        dest="business",
+        default=False,
+        action='store_true',
+        help="Use Whatsapp Business default files (iOS only)"
+    )
+    parser.add_argument(
+        "--preserve-timestamp",
+        dest="preserve_timestamp",
+        default=False,
+        action='store_true',
+        help="Preserve the modification timestamp of the extracted files (iOS only)"
+    )
+    parser.add_argument(
+        "--wab",
+        "--wa-backup",
+        dest="wab",
+        default=None,
+        help="Path to contact database in crypt15 format"
+    )
+    parser.add_argument(
+        "--time-offset",
+        dest="timezone_offset",
+        default=0,
+        type=int,
+        choices=range(-12, 15),
+        metavar="{-12 to 14}",
+        help="Offset in hours (-12 to 14) for time displayed in the output"
+    )
+    parser.add_argument(
+        "--date",
+        dest="filter_date",
+        default=None,
+        metavar="DATE",
+        help="The date filter in specific format (inclusive)"
+    )
+    parser.add_argument(
+        "--date-format",
+        dest="filter_date_format",
+        default="%Y-%m-%d %H:%M",
+        metavar="FORMAT",
+        help="The date format for the date filter"
+    )
+    parser.add_argument(
+        "--include",
+        dest="filter_chat_include",
+        nargs='*',
+        metavar="phone number",
+        help="Include chats that match the supplied phone number"
+    )
+    parser.add_argument(
+        "--exclude",
+        dest="filter_chat_exclude",
+        nargs='*',
+        metavar="phone number",
+        help="Exclude chats that match the supplied phone number"
+    )
+    parser.add_argument(
+        "--per-chat",
+        dest="json_per_chat",
+        default=False,
+        action='store_true',
+        help="Output the JSON file per chat"
+    )
+    parser.add_argument(
+        "--create-separated-media",
+        dest="separate_media",
+        default=False,
+        action='store_true',
+        help="Create a copy of the media seperated per chat in <MEDIA>/separated/ directory"
+    )
     args = parser.parse_args()
 
     # Check for updates
@@ -184,34 +260,80 @@ def main():
         exit(check_update())
 
     # Sanity checks
-    if args.android and args.iphone and args.exported and args.import_json:
-        print("You must define only one device type.")
-        exit(1)
-    if not args.android and not args.iphone and not args.exported and not args.import_json:
-        print("You must define the device type.")
-        exit(1)
+    if args.android and args.ios and args.exported and args.import_json:
+        parser.error("You must define only one device type.")
+    if not args.android and not args.ios and not args.exported and not args.import_json:
+        parser.error("You must define the device type.")
     if args.no_html and not args.json:
-        print("You must either specify a JSON output file or enable HTML output.")
-        exit(1)
-    if args.import_json and (args.android or args.iphone or args.exported or args.no_html):
-        print("You can only use --import with -j and without --no-html.")
-        exit(1)
+        parser.error("You must either specify a JSON output file or enable HTML output.")
+    if args.import_json and (args.android or args.ios or args.exported or args.no_html):
+        parser.error("You can only use --import with -j and without --no-html.")
     elif args.import_json and not os.path.isfile(args.json):
-        print("JSON file not found.")
-        exit(1)
+        parser.error("JSON file not found.")
+    if args.android and args.business:
+        parser.error("WhatsApp Business is only available on iOS for now.")
+    if args.json_per_chat and (
+        (args.json[-5:] != ".json" and os.path.isfile(args.json)) or \
+        (args.json[-5:] == ".json" and os.path.isfile(args.json[:-5]))
+    ):
+        parser.error("When --per-chat is enabled, the destination of --json must be a directory.")
+    if args.filter_date is not None:
+        if " - " in args.filter_date:
+            start, end = args.filter_date.split(" - ")
+            start = int(datetime.strptime(start, args.filter_date_format).timestamp())
+            end = int(datetime.strptime(end, args.filter_date_format).timestamp())
+            if start < 1009843200 or end < 1009843200:
+                parser.error("WhatsApp was first released in 2009...")
+            if start > end:
+                parser.error("The start date cannot be a moment after the end date.")
+            if args.android:
+                args.filter_date = f"BETWEEN {start}000 AND {end}000"
+            elif args.ios:
+                args.filter_date = f"BETWEEN {start - APPLE_TIME} AND {end - APPLE_TIME}"
+        else:
+            _timestamp = int(datetime.strptime(args.filter_date[2:], args.filter_date_format).timestamp())
+            if _timestamp < 1009843200:
+                parser.error("WhatsApp was first released in 2009...")
+            if args.filter_date[:2] == "> ":
+                if args.android:
+                    args.filter_date = f">= {_timestamp}000"
+                elif args.ios:
+                    args.filter_date = f">= {_timestamp - APPLE_TIME}"
+            elif args.filter_date[:2] == "< ":
+                if args.android:
+                    args.filter_date = f"<= {_timestamp}000"
+                elif args.ios:
+                    args.filter_date = f"<= {_timestamp - APPLE_TIME}"
+            else:
+                parser.error("Unsupported date format. See https://wts.knugi.dev/docs?dest=date")
+    if args.filter_chat_include is not None and args.filter_chat_exclude is not None:
+        parser.error("Chat inclusion and exclusion filters cannot be used together.")
+    if args.filter_chat_include is not None:
+        for chat in args.filter_chat_include:
+            if not chat.isnumeric():
+                parser.error("Enter a phone number in the chat filter. See https://wts.knugi.dev/docs?dest=chat")
+    if args.filter_chat_exclude is not None:
+        for chat in args.filter_chat_exclude:
+            if not chat.isnumeric():
+                parser.error("Enter a phone number in the chat filter. See https://wts.knugi.dev/docs?dest=chat")
+    filter_chat = (args.filter_chat_include, args.filter_chat_exclude)
 
     data = {}
 
     if args.android:
-        contacts = extract.contacts
-        messages = extract.messages
-        media = extract.media
-        vcard = extract.vcard
-        create_html = extract.create_html
+        contacts = android_handler.contacts
+        messages = android_handler.messages
+        media = android_handler.media
+        vcard = android_handler.vcard
+        create_html = android_handler.create_html
         if args.db is None:
             msg_db = "msgstore.db"
         else:
             msg_db = args.db
+        if args.wa is None:
+            contact_db = "wa.db"
+        else:
+            contact_db = args.wa
         if args.key is not None:
             if args.backup is None:
                 print("You must specify the backup file with -b")
@@ -228,7 +350,20 @@ def main():
             elif all(char in string.hexdigits for char in args.key):
                 key = bytes.fromhex(args.key)
             db = open(args.backup, "rb").read()
-            error = extract.decrypt_backup(db, key, msg_db, crypt, args.showkey)
+            if args.wab:
+                wab = open(args.wab, "rb").read()
+                error_wa = android_handler.decrypt_backup(wab, key, contact_db, crypt, args.showkey, DbType.CONTACT)
+                if isinstance(key, io.IOBase):
+                    key.seek(0)
+            else:
+                error_wa = 0
+            error_message = android_handler.decrypt_backup(db, key, msg_db, crypt, args.showkey, DbType.MESSAGE)               
+            if error_wa != 0:
+                error = error_wa
+            elif error_message != 0:
+                error = error_message
+            else:
+                error = 0
             if error != 0:
                 if error == 1:
                     print("Dependencies of decrypt_backup and/or extract_encrypted_key"
@@ -241,10 +376,6 @@ def main():
                 else:
                     print("Unknown error occurred.", error)
                     exit(5)
-        if args.wa is None:
-            contact_db = "wa.db"
-        else:
-            contact_db = args.wa
         if args.media is None:
             args.media = "WhatsApp"
 
@@ -252,27 +383,31 @@ def main():
             with sqlite3.connect(contact_db) as db:
                 db.row_factory = sqlite3.Row
                 contacts(db, data)
-    elif args.iphone:
+    elif args.ios:
         import sys
         if "--iphone" in sys.argv:
             print(
                 "WARNING: The --iphone flag is deprecated and will"
                 "be removed in the future. Use --ios instead."
             )
-        contacts = extract_iphone.contacts
-        messages = extract_iphone.messages
-        media = extract_iphone.media
-        vcard = extract_iphone.vcard
-        create_html = extract.create_html
+        contacts = ios_handler.contacts
+        messages = ios_handler.messages
+        media = ios_handler.media
+        vcard = ios_handler.vcard
+        create_html = android_handler.create_html
+        if args.business:
+            from Whatsapp_Chat_Exporter.utility import WhatsAppBusinessIdentifier as identifiers
+        else:
+            from Whatsapp_Chat_Exporter.utility import WhatsAppIdentifier as identifiers
         if args.media is None:
-            args.media = "AppDomainGroup-group.net.whatsapp.WhatsApp.shared"
+            args.media = identifiers.DOMAIN
         if args.backup is not None:
             if not os.path.isdir(args.media):
-                extract_iphone_media.extract_media(args.backup)
+                ios_media_handler.extract_media(args.backup, identifiers, args.preserve_timestamp)
             else:
                 print("WhatsApp directory already exists, skipping WhatsApp file extraction.")
         if args.db is None:
-            msg_db = "7c7fba66680ef796b916b067077cc246adacf01d"
+            msg_db = identifiers.MESSAGE
         else:
             msg_db = args.db
         if args.wa is None:
@@ -288,11 +423,11 @@ def main():
         if os.path.isfile(msg_db):
             with sqlite3.connect(msg_db) as db:
                 db.row_factory = sqlite3.Row
-                messages(db, data, args.media)
-                media(db, data, args.media)
-                vcard(db, data)
+                messages(db, data, args.media, args.timezone_offset, args.filter_date, filter_chat)
+                media(db, data, args.media, args.filter_date, filter_chat, args.separate_media)
+                vcard(db, data, args.media, args.filter_date, filter_chat)
                 if args.android:
-                    extract.calls(db, data)
+                    android_handler.calls(db, data, args.timezone_offset, filter_chat)
             if not args.no_html:
                 create_html(
                     data,
@@ -308,7 +443,7 @@ def main():
                 "The message database does not exist. You may specify the path "
                 "to database file with option -d or check your provided path."
             )
-            exit(2)
+            exit(6)
 
         if os.path.isdir(args.media):
             media_path = os.path.join(args.output, args.media)
@@ -328,9 +463,9 @@ def main():
                         print("\nCannot remove original WhatsApp directory. "
                             "Perhaps the directory is opened?", end="\n")
     elif args.exported:
-        extract_exported.messages(args.exported, data, args.assume_first_as_me)
+        exported_handler.messages(args.exported, data, args.assume_first_as_me)
         if not args.no_html:
-            extract.create_html(
+            android_handler.create_html(
                 data,
                 args.output,
                 args.template,
@@ -342,7 +477,7 @@ def main():
             shutil.copy(file, args.output)
     elif args.import_json:
         import_from_json(args.json, data)
-        extract.create_html(
+        android_handler.create_html(
             data,
             args.output,
             args.template,
@@ -354,10 +489,26 @@ def main():
     if args.json and not args.import_json:
         if isinstance(data[next(iter(data))], ChatStore):
             data = {jik: chat.to_json() for jik, chat in data.items()}
-        with open(args.json, "w") as f:
-            data = json.dumps(data)
-            print(f"\nWriting JSON file...({int(len(data)/1024/1024)}MB)")
-            f.write(data)
+        if not args.json_per_chat:
+            with open(args.json, "w") as f:
+                data = json.dumps(data)
+                print(f"\nWriting JSON file...({int(len(data)/1024/1024)}MB)")
+                f.write(data)
+        else:
+            if args.json[-5:] == ".json":
+                args.json = args.json[:-5]
+            total = len(data.keys())
+            if not os.path.isdir(args.json):
+                os.mkdir(args.json)
+            for index, jik in enumerate(data.keys()):
+                if data[jik]["name"] is not None:
+                    contact = data[jik]["name"].replace('/', '')
+                else:
+                    contact = jik.replace('+', '')
+                with open(f"{args.json}/{contact}.json", "w") as f:
+                    f.write(json.dumps(data[jik]))
+                    print(f"Writing JSON file...({index + 1}/{total})", end="\r")
+            print()
     else:
         print()
 

Whatsapp_Chat_Exporter/android_handler.py

@@ -1,21 +1,20 @@
 #!/usr/bin/python3
 
 import sqlite3
-import json
-import jinja2
 import os
-import shutil
-import re
 import io
 import hmac
+import shutil
 from pathlib import Path
 from mimetypes import MimeTypes
+from markupsafe import escape as htmle
 from hashlib import sha256
 from base64 import b64decode, b64encode
 from Whatsapp_Chat_Exporter.data_model import ChatStore, Message
-from Whatsapp_Chat_Exporter.utility import MAX_SIZE, ROW_SIZE, determine_metadata, get_status_location
+from Whatsapp_Chat_Exporter.utility import MAX_SIZE, ROW_SIZE, DbType, determine_metadata, JidType
 from Whatsapp_Chat_Exporter.utility import rendering, Crypt, Device, get_file_name, setup_template
-from Whatsapp_Chat_Exporter.utility import brute_force_offset, CRYPT14_OFFSETS
+from Whatsapp_Chat_Exporter.utility import brute_force_offset, CRYPT14_OFFSETS, get_status_location
+from Whatsapp_Chat_Exporter.utility import get_chat_condition, slugify
 
 try:
     import zlib
@@ -53,7 +52,7 @@ def _extract_encrypted_key(keyfile):
     return _generate_hmac_of_hmac(key_stream)
 
 
-def decrypt_backup(database, key, output, crypt=Crypt.CRYPT14, show_crypt15=False):
+def decrypt_backup(database, key, output, crypt=Crypt.CRYPT14, show_crypt15=False, db_type=DbType.MESSAGE):
     if not support_backup:
         return 1
     if isinstance(key, io.IOBase):
@@ -83,8 +82,12 @@ def decrypt_backup(database, key, output, crypt=Crypt.CRYPT14, show_crypt15=Fals
         if len(database) < 131:
             raise ValueError("The crypt15 file must be at least 131 bytes")
         t1 = t2 = None
-        iv = database[8:24]
-        db_offset = database[0] + 2  # Skip protobuf + protobuf size and backup type
+        if db_type == DbType.MESSAGE:
+            iv = database[8:24]
+            db_offset = database[0] + 2  # Skip protobuf + protobuf size and backup type
+        elif db_type == DbType.CONTACT:
+            iv = database[7:23]
+            db_offset = database[0] + 1  # Skip protobuf + protobuf size
         db_ciphertext = database[db_offset:]
 
     if t1 != t2:
@@ -165,19 +168,33 @@ def contacts(db, data):
         row = c.fetchone()
 
 
-def messages(db, data, media_folder):
+def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
     # Get message history
     c = db.cursor()
     try:
-        c.execute("""SELECT count() FROM messages""")
+        c.execute(f"""SELECT count()
+                      FROM messages
+                      WHERE 1=1
+                        {f'AND timestamp {filter_date}' if filter_date is not None else ''}
+                        {get_chat_condition(filter_chat[0], True, "messages.key_remote_jid")}
+                        {get_chat_condition(filter_chat[1], False, "messages.key_remote_jid")}""")
+
     except sqlite3.OperationalError:
-        c.execute("""SELECT count() FROM message""")
+        c.execute(f"""SELECT count()
+                      FROM message
+                        LEFT JOIN chat
+                            ON chat._id = message.chat_row_id
+                        INNER JOIN jid
+                            ON jid._id = chat.jid_row_id
+                      WHERE 1=1
+                        {f'AND timestamp {filter_date}' if filter_date is not None else ''}
+                        {get_chat_condition(filter_chat[0], True, "jid.raw_string")}
+                        {get_chat_condition(filter_chat[1], False, "jid.raw_string")}""")
     total_row_number = c.fetchone()[0]
     print(f"Processing messages...(0/{total_row_number})", end="\r")
 
-    phone_number_re = re.compile(r"[0-9]+@s.whatsapp.net")
     try:
-        c.execute("""SELECT messages.key_remote_jid,
+        c.execute(f"""SELECT messages.key_remote_jid,
                             messages._id,
                             messages.key_from_me,
                             messages.timestamp,
@@ -193,12 +210,18 @@ def messages(db, data, media_folder):
                             messages.key_id,
                             messages_quotes.data as quoted_data,
                             messages.media_caption,
-							missed_call_logs.video_call,
+                            missed_call_logs.video_call,
                             chat.subject as chat_subject,
                             message_system.action_type,
                             message_system_group.is_me_joined,
                             jid_old.raw_string as old_jid,
-                            jid_new.raw_string as new_jid
+                            jid_new.raw_string as new_jid,
+                            jid_global.type as jid_type,
+                            group_concat(receipt_user.receipt_timestamp) as receipt_timestamp,
+                            group_concat(messages.received_timestamp) as received_timestamp,
+                            group_concat(receipt_user.read_timestamp) as read_timestamp,
+                            group_concat(receipt_user.played_timestamp) as played_timestamp,
+                            group_concat(messages.read_device_timestamp) as read_device_timestamp
                     FROM messages
                         LEFT JOIN messages_quotes
                             ON messages.quoted_row_id = messages_quotes._id
@@ -218,11 +241,18 @@ def messages(db, data, media_folder):
                             ON jid_old._id = message_system_number_change.old_jid_row_id
                         LEFT JOIN jid jid_new
                             ON jid_new._id = message_system_number_change.new_jid_row_id
-                    WHERE messages.key_remote_jid <> '-1';"""
+                        LEFT JOIN receipt_user
+                            ON receipt_user.message_row_id = messages._id
+                    WHERE messages.key_remote_jid <> '-1'
+                        {f'AND messages.timestamp {filter_date}' if filter_date is not None else ''}
+                        {get_chat_condition(filter_chat[0], True, "messages.key_remote_jid")}
+                        {get_chat_condition(filter_chat[1], False, "messages.key_remote_jid")}
+                    GROUP BY messages._id
+                    ORDER BY messages.timestamp ASC;"""
         )
     except sqlite3.OperationalError:
         try:
-            c.execute("""SELECT jid_global.raw_string as key_remote_jid,
+            c.execute(f"""SELECT jid_global.raw_string as key_remote_jid,
                             message._id,
                             message.from_me as key_from_me,
                             message.timestamp,
@@ -244,7 +274,12 @@ def messages(db, data, media_folder):
                             message_system.action_type,
                             message_system_group.is_me_joined,
                             jid_old.raw_string as old_jid,
-                            jid_new.raw_string as new_jid
+                            jid_new.raw_string as new_jid,
+                            jid_global.type as jid_type,
+                            group_concat(receipt_user.receipt_timestamp) as receipt_timestamp,
+                            group_concat(message.received_timestamp) as received_timestamp,
+                            group_concat(receipt_user.read_timestamp) as read_timestamp,
+                            group_concat(receipt_user.played_timestamp) as played_timestamp
                     FROM message
                         LEFT JOIN message_quoted
                             ON message_quoted.message_row_id = message._id
@@ -274,7 +309,13 @@ def messages(db, data, media_folder):
                             ON jid_old._id = message_system_number_change.old_jid_row_id
                         LEFT JOIN jid jid_new
                             ON jid_new._id = message_system_number_change.new_jid_row_id
-                        WHERE key_remote_jid <> '-1';"""
+                        LEFT JOIN receipt_user
+                            ON receipt_user.message_row_id = message._id
+                    WHERE key_remote_jid <> '-1'
+                        {f'AND message.timestamp {filter_date}' if filter_date is not None else ''}
+                        {get_chat_condition(filter_chat[0], True, "key_remote_jid")}
+                        {get_chat_condition(filter_chat[1], False, "key_remote_jid")}
+                    GROUP BY message._id;"""
             )
         except Exception as e:
             raise e
@@ -304,6 +345,7 @@ def messages(db, data, media_folder):
             timestamp=content["timestamp"],
             time=content["timestamp"],
             key_id=content["key_id"],
+            timezone_offset=timezone_offset
         )
         if isinstance(content["data"], bytes):
             message.data = ("The message is binary data and its base64 is "
@@ -316,7 +358,7 @@ def messages(db, data, media_folder):
             i += 1
             content = c.fetchone()
             continue
-        if "-" in content["key_remote_jid"] and content["key_from_me"] == 0:
+        if content["jid_type"] == JidType.GROUP and content["key_from_me"] == 0:
             name = fallback = None
             if table_message:
                 if content["sender_jid_row_id"] > 0:
@@ -390,8 +432,7 @@ def messages(db, data, media_folder):
                     message.data = None
         else:
             # Real message
-            if content["media_wa_type"] == 20: # Sticker is a message
-                message.sticker = True
+            message.sticker = content["media_wa_type"] == 20  # Sticker is a message
             if content["key_from_me"] == 1:
                 if content["status"] == 5 and content["edit_version"] == 7 or table_message and content["media_wa_type"] == 15:
                     msg = "Message deleted"
@@ -438,15 +479,36 @@ def messages(db, data, media_folder):
     print(f"Processing messages...({total_row_number}/{total_row_number})", end="\r")
 
 
-def media(db, data, media_folder):
+def media(db, data, media_folder, filter_date, filter_chat, separate_media=True):
     # Get media
     c = db.cursor()
-    c.execute("""SELECT count() FROM message_media""")
+    try:
+        c.execute(f"""SELECT count()
+                    FROM message_media
+                        INNER JOIN messages
+                            ON message_media.message_row_id = messages._id
+                    WHERE 1=1  
+                        {f'AND messages.timestamp {filter_date}' if filter_date is not None else ''}
+                        {get_chat_condition(filter_chat[0], True, "messages.key_remote_jid")}
+                        {get_chat_condition(filter_chat[1], False, "messages.key_remote_jid")}""")
+    except sqlite3.OperationalError:
+        c.execute(f"""SELECT count()
+                    FROM message_media
+                        INNER JOIN message
+                            ON message_media.message_row_id = message._id
+                        LEFT JOIN chat
+                            ON chat._id = message.chat_row_id
+                        INNER JOIN jid
+                            ON jid._id = chat.jid_row_id
+                    WHERE 1=1    
+                        {f'AND message.timestamp {filter_date}' if filter_date is not None else ''}
+                        {get_chat_condition(filter_chat[0], True, "jid.raw_string")}
+                        {get_chat_condition(filter_chat[1], False, "jid.raw_string")}""")
     total_row_number = c.fetchone()[0]
     print(f"\nProcessing media...(0/{total_row_number})", end="\r")
     i = 0
     try:
-        c.execute("""SELECT messages.key_remote_jid,
+        c.execute(f"""SELECT messages.key_remote_jid,
                         message_row_id,
                         file_path,
                         message_url,
@@ -459,10 +521,16 @@ def media(db, data, media_folder):
                         ON message_media.message_row_id = messages._id
 					LEFT JOIN media_hash_thumbnail
 						ON message_media.file_hash = media_hash_thumbnail.media_hash
+                    INNER JOIN jid
+                        ON messages.key_remote_jid = jid.raw_string
+                WHERE jid.type <> 7
+                    {f'AND messages.timestamp {filter_date}' if filter_date is not None else ''}
+                    {get_chat_condition(filter_chat[0], True, "messages.key_remote_jid")}
+                    {get_chat_condition(filter_chat[1], False, "messages.key_remote_jid")}
                 ORDER BY messages.key_remote_jid ASC"""
         )
     except sqlite3.OperationalError:
-        c.execute("""SELECT jid.raw_string as key_remote_jid,
+        c.execute(f"""SELECT jid.raw_string as key_remote_jid,
                     message_row_id,
                     file_path,
                     message_url,
@@ -471,14 +539,18 @@ def media(db, data, media_folder):
                     file_hash,
                     thumbnail
                 FROM message_media
-                INNER JOIN message
-                    ON message_media.message_row_id = message._id
-                LEFT JOIN chat
-                    ON chat._id = message.chat_row_id
-                INNER JOIN jid
-                    ON jid._id = chat.jid_row_id
-                LEFT JOIN media_hash_thumbnail
+                    INNER JOIN message
+                        ON message_media.message_row_id = message._id
+                    LEFT JOIN chat
+                        ON chat._id = message.chat_row_id
+                    INNER JOIN jid
+                        ON jid._id = chat.jid_row_id
+                    LEFT JOIN media_hash_thumbnail
 						ON message_media.file_hash = media_hash_thumbnail.media_hash
+                WHERE jid.type <> 7
+                    {f'AND message.timestamp {filter_date}' if filter_date is not None else ''}
+                    {get_chat_condition(filter_chat[0], True, "key_remote_jid")}
+                    {get_chat_condition(filter_chat[1], False, "key_remote_jid")}
                 ORDER BY jid.raw_string ASC"""
         )
     content = c.fetchone()
@@ -499,6 +571,15 @@ def media(db, data, media_folder):
                     message.mime = "application/octet-stream"
             else:
                 message.mime = content["mime_type"]
+            if separate_media:
+                chat_display_name = slugify(data[content["key_remote_jid"]].name or message.sender \
+                                            or content["key_remote_jid"].split('@')[0], True)
+                current_filename = file_path.split("/")[-1]
+                new_folder = os.path.join(media_folder, "separated", chat_display_name)
+                Path(new_folder).mkdir(parents=True, exist_ok=True)
+                new_path = os.path.join(new_folder, current_filename)
+                shutil.copy2(file_path, new_path)
+                message.data = new_path
         else:
             if False: # Block execution
                 try:
@@ -528,20 +609,24 @@ def media(db, data, media_folder):
         f"Processing media...({total_row_number}/{total_row_number})", end="\r")
 
 
-def vcard(db, data):
+def vcard(db, data, media_folder, filter_date, filter_chat):
     c = db.cursor()
     try:
-        c.execute("""SELECT message_row_id,
+        c.execute(f"""SELECT message_row_id,
                         messages.key_remote_jid,
                         vcard,
                         messages.media_name
                  FROM messages_vcards
                     INNER JOIN messages
                         ON messages_vcards.message_row_id = messages._id
+                 WHERE 1=1
+                    {f'AND messages.timestamp {filter_date}' if filter_date is not None else ''}
+                    {get_chat_condition(filter_chat[0], True, "messages.key_remote_jid")}
+                    {get_chat_condition(filter_chat[1], False, "messages.key_remote_jid")}
                  ORDER BY messages.key_remote_jid ASC;"""
         )
     except sqlite3.OperationalError:
-        c.execute("""SELECT message_row_id,
+        c.execute(f"""SELECT message_row_id,
                         jid.raw_string as key_remote_jid,
                         vcard,
                         message.text_data as media_name
@@ -552,40 +637,52 @@ def vcard(db, data):
                         ON chat._id = message.chat_row_id
                     INNER JOIN jid
                         ON jid._id = chat.jid_row_id
+                WHERE 1=1
+                    {f'AND message.timestamp {filter_date}' if filter_date is not None else ''}
+                    {get_chat_condition(filter_chat[0], True, "key_remote_jid")}
+                    {get_chat_condition(filter_chat[1], False, "key_remote_jid")}
                  ORDER BY message.chat_row_id ASC;"""
         )
 
     rows = c.fetchall()
     total_row_number = len(rows)
     print(f"\nProcessing vCards...(0/{total_row_number})", end="\r")
-    base = "WhatsApp/vCards"
-    if not os.path.isdir(base):
-        Path(base).mkdir(parents=True, exist_ok=True)
+    path = f"{media_folder}/vCards"
+    if not os.path.isdir(path):
+        Path(path).mkdir(parents=True, exist_ok=True)
     for index, row in enumerate(rows):
-        media_name = row["media_name"] if row["media_name"] is not None else ""
+        media_name = row["media_name"] if row["media_name"] is not None else "Undefined vCard File"
         file_name = "".join(x for x in media_name if x.isalnum())
         file_name = file_name.encode('utf-8')[:230].decode('utf-8', 'ignore')
-        file_path = os.path.join(base, f"{file_name}.vcf")
+        file_path = os.path.join(path, f"{file_name}.vcf")
         if not os.path.isfile(file_path):
             with open(file_path, "w", encoding="utf-8") as f:
                 f.write(row["vcard"])
         message = data[row["key_remote_jid"]].messages[row["message_row_id"]]
-        message.data = media_name + \
-            "The vCard file cannot be displayed here, " \
-            f"however it should be located at {file_path}"
+        message.data = "This media include the following vCard file(s):<br>" \
+            f'<a href="{htmle(file_path)}">{htmle(media_name)}</a>'
         message.mime = "text/x-vcard"
         message.meta = True
+        message.safe = True
         print(f"Processing vCards...({index + 1}/{total_row_number})", end="\r")
 
 
-def calls(db, data):
+def calls(db, data, timezone_offset, filter_chat):
     c = db.cursor()
-    c.execute("""SELECT count() FROM call_log""")
+    c.execute(f"""SELECT count()
+                FROM call_log
+                    INNER JOIN jid
+                        ON call_log.jid_row_id = jid._id
+                    LEFT JOIN chat
+                        ON call_log.jid_row_id = chat.jid_row_id
+                WHERE 1=1
+                    {get_chat_condition(filter_chat[0], True, "jid.raw_string")}
+                    {get_chat_condition(filter_chat[1], False, "jid.raw_string")}""")
     total_row_number = c.fetchone()[0]
     if total_row_number == 0:
         return
     print(f"\nProcessing calls...({total_row_number})", end="\r")
-    c.execute("""SELECT call_log._id,
+    c.execute(f"""SELECT call_log._id,
                         jid.raw_string,
                         from_me,
                         call_id,
@@ -593,10 +690,16 @@ def calls(db, data):
                         video_call,
                         duration,
                         call_result,
-                        bytes_transferred
+                        bytes_transferred,
+                        chat.subject as chat_subject
                 FROM call_log
                     INNER JOIN jid
-                        ON call_log.jid_row_id = jid._id"""
+                        ON call_log.jid_row_id = jid._id
+                    LEFT JOIN chat
+                        ON call_log.jid_row_id = chat.jid_row_id
+                WHERE 1=1
+                    {get_chat_condition(filter_chat[0], True, "jid.raw_string")}
+                    {get_chat_condition(filter_chat[1], False, "jid.raw_string")}"""
     )
     chat = ChatStore(Device.ANDROID, "WhatsApp Calls")
     content = c.fetchone()
@@ -606,18 +709,20 @@ def calls(db, data):
             timestamp=content["timestamp"],
             time=content["timestamp"],
             key_id=content["call_id"],
+            timezone_offset=timezone_offset
         )
         _jid = content["raw_string"]
-        if _jid in data:
-            name = data[_jid].name
-            fallback = _jid.split('@')[0] if "@" in _jid else None
-            call.sender = name or fallback
-
+        name = data[_jid].name if _jid in data else content["chat_subject"] or None
+        if _jid is not None and "@" in _jid:
+            fallback = _jid.split('@')[0]
+        else:
+            fallback = None
+        call.sender = name or fallback
         call.meta = True
         call.data = (
             f"A {'video' if content['video_call'] else 'voice'} "
             f"call {'to' if call.from_me else 'from'} "
-            f"{name or fallback} was "
+            f"{call.sender} was "
         )
         if content['call_result'] == 2:
             call.data += "not answered." if call.from_me else "missed."

Whatsapp_Chat_Exporter/bplist.py

@@ -0,0 +1,292 @@
+#################################################################################
+# Copyright (C) 2009-2011 Vladimir "Farcaller" Pouzanov <farcaller@gmail.com>   #
+#                                                                               #
+# Permission is hereby granted, free of charge, to any person obtaining a copy  #
+# of this software and associated documentation files (the "Software"), to deal #
+# in the Software without restriction, including without limitation the rights  #
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell     #
+# copies of the Software, and to permit persons to whom the Software is         #
+# furnished to do so, subject to the following conditions:                      #
+#                                                                               #
+# The above copyright notice and this permission notice shall be included in    #
+# all copies or substantial portions of the Software.                           #
+#                                                                               #
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR    #
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,      #
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE   #
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER        #
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, #
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN     #
+# THE SOFTWARE.                                                                 #
+#################################################################################
+
+import struct
+import codecs
+from datetime import datetime, timedelta
+
+class BPListWriter(object):
+    def __init__(self, objects):
+        self.bplist = ""
+        self.objects = objects
+    
+    def binary(self):
+        '''binary -> string
+        
+        Generates bplist
+        '''
+        self.data = 'bplist00'
+        
+        # TODO: flatten objects and count max length size
+        
+        # TODO: write objects and save offsets
+        
+        # TODO: write offsets
+        
+        # TODO: write metadata
+        
+        return self.data
+    
+    def write(self, filename):
+        '''
+        
+        Writes bplist to file
+        '''
+        if self.bplist != "":
+            pass
+            # TODO: save self.bplist to file
+        else:
+            raise Exception('BPlist not yet generated')
+
+class BPListReader(object):
+    def __init__(self, s):
+        self.data = s
+        self.objects = []
+        self.resolved = {}
+    
+    def __unpackIntStruct(self, sz, s):
+        '''__unpackIntStruct(size, string) -> int
+        
+        Unpacks the integer of given size (1, 2 or 4 bytes) from string
+        '''
+        if   sz == 1:
+            ot = '!B'
+        elif sz == 2:
+            ot = '!H'
+        elif sz == 4:
+            ot = '!I'
+        elif sz == 8:
+            ot = '!Q'
+        else:
+            raise Exception('int unpack size '+str(sz)+' unsupported')
+        return struct.unpack(ot, s)[0]
+    
+    def __unpackInt(self, offset):
+        '''__unpackInt(offset) -> int
+        
+        Unpacks int field from plist at given offset
+        '''
+        return self.__unpackIntMeta(offset)[1]
+
+    def __unpackIntMeta(self, offset):
+        '''__unpackIntMeta(offset) -> (size, int)
+        
+        Unpacks int field from plist at given offset and returns its size and value
+        '''
+        obj_header = self.data[offset]
+        obj_type, obj_info = (obj_header & 0xF0), (obj_header & 0x0F)
+        int_sz = 2**obj_info
+        return int_sz, self.__unpackIntStruct(int_sz, self.data[offset+1:offset+1+int_sz])
+
+    def __resolveIntSize(self, obj_info, offset):
+        '''__resolveIntSize(obj_info, offset) -> (count, offset)
+        
+        Calculates count of objref* array entries and returns count and offset to first element
+        '''
+        if obj_info == 0x0F:
+            ofs, obj_count = self.__unpackIntMeta(offset+1)
+            objref = offset+2+ofs
+        else:
+            obj_count = obj_info
+            objref = offset+1
+        return obj_count, objref
+
+    def __unpackFloatStruct(self, sz, s):
+        '''__unpackFloatStruct(size, string) -> float
+        
+        Unpacks the float of given size (4 or 8 bytes) from string
+        '''
+        if   sz == 4:
+            ot = '!f'
+        elif sz == 8:
+            ot = '!d'
+        else:
+            raise Exception('float unpack size '+str(sz)+' unsupported')
+        return struct.unpack(ot, s)[0]
+
+    def __unpackFloat(self, offset):
+        '''__unpackFloat(offset) -> float
+        
+        Unpacks float field from plist at given offset
+        '''
+        obj_header = self.data[offset]
+        obj_type, obj_info = (obj_header & 0xF0), (obj_header & 0x0F)
+        int_sz = 2**obj_info
+        return int_sz, self.__unpackFloatStruct(int_sz, self.data[offset+1:offset+1+int_sz])
+
+    def __unpackDate(self, offset):
+        td = int(struct.unpack(">d", self.data[offset+1:offset+9])[0])
+        return datetime(year=2001,month=1,day=1) + timedelta(seconds=td)
+
+    def __unpackItem(self, offset):
+        '''__unpackItem(offset)
+        
+        Unpacks and returns an item from plist
+        '''
+        obj_header = self.data[offset]
+        obj_type, obj_info = (obj_header & 0xF0), (obj_header & 0x0F)
+        if   obj_type == 0x00:
+            if   obj_info == 0x00: # null   0000 0000
+                return None
+            elif obj_info == 0x08: # bool   0000 1000           // false
+                return False
+            elif obj_info == 0x09: # bool   0000 1001           // true
+                return True
+            elif obj_info == 0x0F: # fill   0000 1111           // fill byte
+                raise Exception("0x0F Not Implemented") # this is really pad byte, FIXME
+            else:
+                raise Exception('unpack item type '+str(obj_header)+' at '+str(offset)+ 'failed')
+        elif obj_type == 0x10: #     int    0001 nnnn   ...     // # of bytes is 2^nnnn, big-endian bytes
+            return self.__unpackInt(offset)
+        elif obj_type == 0x20: #    real    0010 nnnn   ...     // # of bytes is 2^nnnn, big-endian bytes
+            return self.__unpackFloat(offset)
+        elif obj_type == 0x30: #    date    0011 0011   ...     // 8 byte float follows, big-endian bytes
+            return self.__unpackDate(offset)
+        elif obj_type == 0x40: #    data    0100 nnnn   [int]   ... // nnnn is number of bytes unless 1111 then int count follows, followed by bytes
+            obj_count, objref = self.__resolveIntSize(obj_info, offset)
+            return self.data[objref:objref+obj_count] # XXX: we return data as str
+        elif obj_type == 0x50: #    string  0101 nnnn   [int]   ... // ASCII string, nnnn is # of chars, else 1111 then int count, then bytes
+            obj_count, objref = self.__resolveIntSize(obj_info, offset)
+            return self.data[objref:objref+obj_count]
+        elif obj_type == 0x60: #    string  0110 nnnn   [int]   ... // Unicode string, nnnn is # of chars, else 1111 then int count, then big-endian 2-byte uint16_t
+            obj_count, objref = self.__resolveIntSize(obj_info, offset)
+            return self.data[objref:objref+obj_count*2].decode('utf-16be')
+        elif obj_type == 0x80: #    uid     1000 nnnn   ...     // nnnn+1 is # of bytes
+            # FIXME: Accept as a string for now
+            obj_count, objref = self.__resolveIntSize(obj_info, offset)
+            return self.data[objref:objref+obj_count]
+        elif obj_type == 0xA0: #    array   1010 nnnn   [int]   objref* // nnnn is count, unless '1111', then int count follows
+            obj_count, objref = self.__resolveIntSize(obj_info, offset)
+            arr = []
+            for i in range(obj_count):
+                arr.append(self.__unpackIntStruct(self.object_ref_size, self.data[objref+i*self.object_ref_size:objref+i*self.object_ref_size+self.object_ref_size]))
+            return arr
+        elif obj_type == 0xC0: #   set      1100 nnnn   [int]   objref* // nnnn is count, unless '1111', then int count follows
+            # XXX: not serializable via apple implementation
+            raise Exception("0xC0 Not Implemented") # FIXME: implement
+        elif obj_type == 0xD0: #   dict     1101 nnnn   [int]   keyref* objref* // nnnn is count, unless '1111', then int count follows
+            obj_count, objref = self.__resolveIntSize(obj_info, offset)
+            keys = []
+            for i in range(obj_count):
+                keys.append(self.__unpackIntStruct(self.object_ref_size, self.data[objref+i*self.object_ref_size:objref+i*self.object_ref_size+self.object_ref_size]))
+            values = []
+            objref += obj_count*self.object_ref_size
+            for i in range(obj_count):
+                values.append(self.__unpackIntStruct(self.object_ref_size, self.data[objref+i*self.object_ref_size:objref+i*self.object_ref_size+self.object_ref_size]))
+            dic = {}
+            for i in range(obj_count):
+                dic[keys[i]] = values[i]
+            return dic
+        else:
+            raise Exception('don\'t know how to unpack obj type '+hex(obj_type)+' at '+str(offset))
+    
+    def __resolveObject(self, idx):
+        try:
+            return self.resolved[idx]
+        except KeyError:
+            obj = self.objects[idx]
+            if type(obj) == list:
+                newArr = []
+                for i in obj:
+                    newArr.append(self.__resolveObject(i))
+                self.resolved[idx] = newArr
+                return newArr
+            if type(obj) == dict:
+                newDic = {}
+                for k,v in obj.items():
+                    key_resolved = self.__resolveObject(k)
+                    if isinstance(key_resolved, str):
+                        rk = key_resolved
+                    else:
+                        rk = codecs.decode(key_resolved, "utf-8")
+                    rv = self.__resolveObject(v)
+                    newDic[rk] = rv
+                self.resolved[idx] = newDic
+                return newDic
+            else:
+                self.resolved[idx] = obj
+                return obj
+    
+    def parse(self):
+        # read header
+        if self.data[:8] != b'bplist00':
+            raise Exception('Bad magic')
+        
+        # read trailer
+        self.offset_size, self.object_ref_size, self.number_of_objects, self.top_object, self.table_offset = struct.unpack('!6xBB4xI4xI4xI', self.data[-32:])
+        #print "** plist offset_size:",self.offset_size,"objref_size:",self.object_ref_size,"num_objs:",self.number_of_objects,"top:",self.top_object,"table_ofs:",self.table_offset
+        
+        # read offset table
+        self.offset_table = self.data[self.table_offset:-32]
+        self.offsets = []
+        ot = self.offset_table
+        for i in range(self.number_of_objects):
+            offset_entry = ot[:self.offset_size]
+            ot = ot[self.offset_size:]
+            self.offsets.append(self.__unpackIntStruct(self.offset_size, offset_entry))
+        #print "** plist offsets:",self.offsets
+        
+        # read object table
+        self.objects = []
+        k = 0
+        for i in self.offsets:
+            obj = self.__unpackItem(i)
+            #print "** plist unpacked",k,type(obj),obj,"at",i
+            k += 1
+            self.objects.append(obj)
+        
+        # rebuild object tree
+        #for i in range(len(self.objects)):
+        #    self.__resolveObject(i)
+        
+        # return root object
+        return self.__resolveObject(self.top_object)
+    
+    @classmethod
+    def plistWithString(cls, s):
+        parser = cls(s)
+        return parser.parse()
+
+# helpers for testing
+def plist(obj):
+    from Foundation import NSPropertyListSerialization, NSPropertyListBinaryFormat_v1_0
+    b = NSPropertyListSerialization.dataWithPropertyList_format_options_error_(obj,  NSPropertyListBinaryFormat_v1_0, 0, None)
+    return str(b.bytes())
+
+def unplist(s):
+    from Foundation import NSData, NSPropertyListSerialization
+    d = NSData.dataWithBytes_length_(s, len(s))
+    return NSPropertyListSerialization.propertyListWithData_options_format_error_(d, 0, None, None)
+
+if __name__ == "__main__":
+    import os
+    import sys
+    import json
+    file_path = sys.argv[1]
+
+    with open(file_path, "rb") as fp:
+        data = fp.read()
+
+    out = BPListReader(data).parse()
+
+    with open(file_path + ".json", "w") as fp:
+        json.dump(out, indent=4)

Whatsapp_Chat_Exporter/data_model.py

@@ -1,10 +1,19 @@
 #!/usr/bin/python3
 
 import os
-from datetime import datetime
+from datetime import datetime, tzinfo, timedelta
 from typing import Union
 
 
+class TimeZone(tzinfo):
+    def __init__(self, offset):
+        self.offset = offset
+    def utcoffset(self, dt):
+       return timedelta(hours=self.offset)
+    def dst(self, dt):
+       return timedelta(0)
+
+
 class ChatStore():
     def __init__(self, type, name=None, media=None):
         if name is not None and not isinstance(name, str):
@@ -55,15 +64,15 @@ class ChatStore():
 
 
 class Message():
-    def __init__(self, from_me: Union[bool,int], timestamp: int, time: Union[int,float,str], key_id: int):
+    def __init__(self, from_me: Union[bool,int], timestamp: int, time: Union[int,float,str], key_id: int, timezone_offset: int = 0):
         self.from_me = bool(from_me)
         self.timestamp = timestamp / 1000 if timestamp > 9999999999 else timestamp
         if isinstance(time, int) or isinstance(time, float):
-            self.time = datetime.fromtimestamp(time/1000).strftime("%H:%M")
+            self.time = datetime.fromtimestamp(self.timestamp, TimeZone(timezone_offset)).strftime("%H:%M")
         elif isinstance(time, str):
             self.time = time
         else:
-            raise TypeError("Time must be a string or integer")
+            raise TypeError("Time must be a string or number")
         self.media = False
         self.key_id = key_id
         self.meta = False

Whatsapp_Chat_Exporter/ios_handler.py

@@ -1,11 +1,13 @@
 #!/usr/bin/python3
 
 import os
+import shutil
 from glob import glob
 from pathlib import Path
 from mimetypes import MimeTypes
+from markupsafe import escape as htmle
 from Whatsapp_Chat_Exporter.data_model import ChatStore, Message
-from Whatsapp_Chat_Exporter.utility import APPLE_TIME, Device
+from Whatsapp_Chat_Exporter.utility import APPLE_TIME, Device, get_chat_condition, slugify
 
 
 def contacts(db, data):
@@ -18,26 +20,33 @@ def contacts(db, data):
     content = c.fetchone()
     while content is not None:
         if not content["ZWHATSAPPID"].endswith("@s.whatsapp.net"):
-            _id = content["ZWHATSAPPID"] + "@s.whatsapp.net"
-        data[_id] = ChatStore(Device.IOS)
-        data[_id].status = content["ZABOUTTEXT"]
+            ZWHATSAPPID = content["ZWHATSAPPID"] + "@s.whatsapp.net"
+        data[ZWHATSAPPID] = ChatStore(Device.IOS)
+        data[ZWHATSAPPID].status = content["ZABOUTTEXT"]
         content = c.fetchone()
 
 
-def messages(db, data, media_folder):
+def messages(db, data, media_folder, timezone_offset, filter_date, filter_chat):
     c = db.cursor()
     # Get contacts
-    c.execute("""SELECT count() FROM ZWACHATSESSION""")
+    c.execute(f"""SELECT count()
+                FROM ZWACHATSESSION
+                WHERE 1=1
+                    {get_chat_condition(filter_chat[0], True, "ZWACHATSESSION.ZCONTACTJID")}
+                    {get_chat_condition(filter_chat[1], False, "ZWACHATSESSION.ZCONTACTJID")}""")
     total_row_number = c.fetchone()[0]
     print(f"Processing contacts...({total_row_number})")
 
     c.execute(
-        """SELECT ZCONTACTJID,
+        f"""SELECT ZCONTACTJID,
                 ZPARTNERNAME,
                 ZPUSHNAME
            FROM ZWACHATSESSION
                 LEFT JOIN ZWAPROFILEPUSHNAME
-                    ON ZWACHATSESSION.ZCONTACTJID = ZWAPROFILEPUSHNAME.ZJID;"""
+                    ON ZWACHATSESSION.ZCONTACTJID = ZWAPROFILEPUSHNAME.ZJID
+           WHERE 1=1
+                {get_chat_condition(filter_chat[0], True, "ZWACHATSESSION.ZCONTACTJID")}
+                {get_chat_condition(filter_chat[1], False, "ZWACHATSESSION.ZCONTACTJID")};"""
     )
     content = c.fetchone()
     while content is not None:
@@ -65,11 +74,17 @@ def messages(db, data, media_folder):
         content = c.fetchone()
 
     # Get message history
-    c.execute("""SELECT count() FROM ZWAMESSAGE""")
+    c.execute(f"""SELECT count()
+                  FROM ZWAMESSAGE
+                        INNER JOIN ZWACHATSESSION
+                            ON ZWAMESSAGE.ZCHATSESSION = ZWACHATSESSION.Z_PK
+                  WHERE 1=1
+                    {f'AND ZMESSAGEDATE {filter_date}' if filter_date is not None else ''}
+                    {get_chat_condition(filter_chat[0], True, "ZWACHATSESSION.ZCONTACTJID")}
+                    {get_chat_condition(filter_chat[1], False, "ZWACHATSESSION.ZCONTACTJID")}""")
     total_row_number = c.fetchone()[0]
     print(f"Processing messages...(0/{total_row_number})", end="\r")
-
-    c.execute("""SELECT COALESCE(ZFROMJID, ZTOJID) as _id,
+    c.execute(f"""SELECT ZCONTACTJID,
                         ZWAMESSAGE.Z_PK,
                         ZISFROMME,
                         ZMESSAGEDATE,
@@ -77,38 +92,48 @@ def messages(db, data, media_folder):
                         ZMESSAGETYPE,
                         ZWAGROUPMEMBER.ZMEMBERJID,
 						ZMETADATA,
-                        ZSTANZAID
+                        ZSTANZAID,
+                        ZGROUPINFO
                  FROM ZWAMESSAGE
                     LEFT JOIN ZWAGROUPMEMBER
                         ON ZWAMESSAGE.ZGROUPMEMBER = ZWAGROUPMEMBER.Z_PK
 					LEFT JOIN ZWAMEDIAITEM
-						ON ZWAMESSAGE.Z_PK = ZWAMEDIAITEM.ZMESSAGE;""")
+						ON ZWAMESSAGE.Z_PK = ZWAMEDIAITEM.ZMESSAGE
+                    INNER JOIN ZWACHATSESSION
+                        ON ZWAMESSAGE.ZCHATSESSION = ZWACHATSESSION.Z_PK
+                 WHERE 1=1   
+                    {f'AND ZMESSAGEDATE {filter_date}' if filter_date is not None else ''}
+                    {get_chat_condition(filter_chat[0], True, "ZCONTACTJID")}
+                    {get_chat_condition(filter_chat[1], False, "ZCONTACTJID")}
+                 ORDER BY ZMESSAGEDATE ASC;""")
     i = 0
     content = c.fetchone()
     while content is not None:
-        _id = content["_id"]
+        ZCONTACTJID = content["ZCONTACTJID"]
         Z_PK = content["Z_PK"]
-        if _id not in data:
-            data[_id] = ChatStore(Device.IOS)
-            path = f'{media_folder}/Media/Profile/{_id.split("@")[0]}'
+        is_group_message = content["ZGROUPINFO"] is not None
+        if ZCONTACTJID not in data:
+            data[ZCONTACTJID] = ChatStore(Device.IOS)
+            path = f'{media_folder}/Media/Profile/{ZCONTACTJID.split("@")[0]}'
             avatars = glob(f"{path}*")
             if 0 < len(avatars) <= 1:
-                data[_id].their_avatar = avatars[0]
+                data[ZCONTACTJID].their_avatar = avatars[0]
             else:
                 for avatar in avatars:
                     if avatar.endswith(".thumb"):
-                        data[_id].their_avatar_thumb = avatar
+                        data[ZCONTACTJID].their_avatar_thumb = avatar
                     elif avatar.endswith(".jpg"):
-                        data[_id].their_avatar = avatar
+                        data[ZCONTACTJID].their_avatar = avatar
         ts = APPLE_TIME + content["ZMESSAGEDATE"]
         message = Message(
             from_me=content["ZISFROMME"],
             timestamp=ts,
             time=ts, # TODO: Could be bug
             key_id=content["ZSTANZAID"][:17],
+            timezone_offset=timezone_offset
         )
         invalid = False
-        if "-" in _id and content["ZISFROMME"] == 0:
+        if is_group_message and content["ZISFROMME"] == 0:
             name = None
             if content["ZMEMBERJID"] is not None:
                 if content["ZMEMBERJID"] in data:
@@ -124,7 +149,7 @@ def messages(db, data, media_folder):
             message.sender = None
         if content["ZMESSAGETYPE"] == 6:
             # Metadata
-            if "-" in _id:
+            if is_group_message:
                 # Group
                 if content["ZTEXT"] is not None:
                     # Chnaged name
@@ -173,7 +198,7 @@ def messages(db, data, media_folder):
                             msg = msg.replace("\n", "<br>")
             message.data = msg
         if not invalid:
-            data[_id].add_message(Z_PK, message)
+            data[ZCONTACTJID].add_message(Z_PK, message)
         i += 1
         if i % 1000 == 0:
             print(f"Processing messages...({i}/{total_row_number})", end="\r")
@@ -182,14 +207,24 @@ def messages(db, data, media_folder):
         f"Processing messages...({total_row_number}/{total_row_number})", end="\r")
 
 
-def media(db, data, media_folder):
+def media(db, data, media_folder, filter_date, filter_chat, separate_media=False):
     c = db.cursor()
     # Get media
-    c.execute("""SELECT count() FROM ZWAMEDIAITEM""")
+    c.execute(f"""SELECT count()
+                  FROM ZWAMEDIAITEM
+                    INNER JOIN ZWAMESSAGE
+                        ON ZWAMEDIAITEM.ZMESSAGE = ZWAMESSAGE.Z_PK
+                    INNER JOIN ZWACHATSESSION
+                        ON ZWAMESSAGE.ZCHATSESSION = ZWACHATSESSION.Z_PK
+                  WHERE 1=1
+                    {f'AND ZMESSAGEDATE {filter_date}' if filter_date is not None else ''}
+                    {get_chat_condition(filter_chat[0], True, "ZWACHATSESSION.ZCONTACTJID")}
+                    {get_chat_condition(filter_chat[1], False, "ZWACHATSESSION.ZCONTACTJID")}
+                """)
     total_row_number = c.fetchone()[0]
     print(f"\nProcessing media...(0/{total_row_number})", end="\r")
     i = 0
-    c.execute("""SELECT COALESCE(ZWAMESSAGE.ZFROMJID, ZWAMESSAGE.ZTOJID) as _id,
+    c.execute(f"""SELECT ZCONTACTJID,
                         ZMESSAGE,
                         ZMEDIALOCALPATH,
                         ZMEDIAURL,
@@ -199,15 +234,19 @@ def media(db, data, media_folder):
                  FROM ZWAMEDIAITEM
                     INNER JOIN ZWAMESSAGE
                         ON ZWAMEDIAITEM.ZMESSAGE = ZWAMESSAGE.Z_PK
+                    INNER JOIN ZWACHATSESSION
+                        ON ZWAMESSAGE.ZCHATSESSION = ZWACHATSESSION.Z_PK
                  WHERE ZMEDIALOCALPATH IS NOT NULL
-                 ORDER BY _id ASC""")
+                    {f'AND ZWAMESSAGE.ZMESSAGEDATE {filter_date}' if filter_date is not None else ''}
+                    {get_chat_condition(filter_chat[0], True, "ZCONTACTJID")}
+                    {get_chat_condition(filter_chat[1], False, "ZCONTACTJID")}
+                 ORDER BY ZCONTACTJID ASC""")
     content = c.fetchone()
     mime = MimeTypes()
     while content is not None:
         file_path = f"{media_folder}/Message/{content['ZMEDIALOCALPATH']}"
-        _id = content["_id"]
         ZMESSAGE = content["ZMESSAGE"]
-        message = data[_id].messages[ZMESSAGE]
+        message = data[content["ZCONTACTJID"]].messages[ZMESSAGE]
         message.media = True
         if os.path.isfile(file_path):
             message.data = file_path
@@ -219,6 +258,15 @@ def media(db, data, media_folder):
                     message.mime = "application/octet-stream"
             else:
                 message.mime = content["ZVCARDSTRING"]
+            if separate_media:
+                chat_display_name = slugify(data[content["ZCONTACTJID"]].name or message.sender \
+                                            or content["ZCONTACTJID"].split('@')[0], True)
+                current_filename = file_path.split("/")[-1]
+                new_folder = os.path.join(media_folder, "separated", chat_display_name)
+                Path(new_folder).mkdir(parents=True, exist_ok=True)
+                new_path = os.path.join(new_folder, current_filename)
+                shutil.copy2(file_path, new_path)
+                message.data = new_path
         else:
             if False: # Block execution
                 try:
@@ -244,37 +292,55 @@ def media(db, data, media_folder):
         f"Processing media...({total_row_number}/{total_row_number})", end="\r")
 
 
-def vcard(db, data):
+def vcard(db, data, media_folder, filter_date, filter_chat):
     c = db.cursor()
-    c.execute("""SELECT DISTINCT ZWAVCARDMENTION.ZMEDIAITEM,
+    c.execute(f"""SELECT DISTINCT ZWAVCARDMENTION.ZMEDIAITEM,
                         ZWAMEDIAITEM.ZMESSAGE,
-                        COALESCE(ZWAMESSAGE.ZFROMJID,
-                        ZWAMESSAGE.ZTOJID) as _id,
+                        ZCONTACTJID,
                         ZVCARDNAME,
                         ZVCARDSTRING
                  FROM ZWAVCARDMENTION
                     INNER JOIN ZWAMEDIAITEM
                         ON ZWAVCARDMENTION.ZMEDIAITEM = ZWAMEDIAITEM.Z_PK
                     INNER JOIN ZWAMESSAGE
-                        ON ZWAMEDIAITEM.ZMESSAGE = ZWAMESSAGE.Z_PK""")
+                        ON ZWAMEDIAITEM.ZMESSAGE = ZWAMESSAGE.Z_PK
+                    INNER JOIN ZWACHATSESSION
+                        ON ZWAMESSAGE.ZCHATSESSION = ZWACHATSESSION.Z_PK
+                 WHERE 1=1
+                    {f'AND ZWAMESSAGE.ZMESSAGEDATE {filter_date}' if filter_date is not None else ''}
+                    {get_chat_condition(filter_chat[0], True, "ZCONTACTJID")}
+                    {get_chat_condition(filter_chat[1], False, "ZCONTACTJID")};""")
     contents = c.fetchall()
     total_row_number = len(contents)
     print(f"\nProcessing vCards...(0/{total_row_number})", end="\r")
-    base = "AppDomainGroup-group.net.whatsapp.WhatsApp.shared/Message/vCards"
-    if not os.path.isdir(base):
-        Path(base).mkdir(parents=True, exist_ok=True)
+    path = f'{media_folder}/Message/vCards'
+    Path(path).mkdir(parents=True, exist_ok=True)
+
     for index, content in enumerate(contents):
-        file_name = "".join(x for x in content["ZVCARDNAME"] if x.isalnum())
-        file_name = file_name.encode('utf-8')[:230].decode('utf-8', 'ignore')
-        file_path = os.path.join(base, f"{file_name}.vcf")
-        if not os.path.isfile(file_path):
-            with open(file_path, "w", encoding="utf-8") as f:
-                f.write(content["ZVCARDSTRING"])
-        message = data[content["_id"]].messages[content["ZMESSAGE"]]
-        message.data = content["ZVCARDNAME"] + \
-            "The vCard file cannot be displayed here, " \
-            f"however it should be located at {file_path}"
+        file_paths = []
+        vcard_names = content["ZVCARDNAME"].split("_$!<Name-Separator>!$_")
+        vcard_strings = content["ZVCARDSTRING"].split("_$!<VCard-Separator>!$_")
+
+        # If this is a list of contacts
+        if len(vcard_names) > len(vcard_strings):
+            vcard_names.pop(0)  # Dismiss the first element, which is the group name
+
+        for name, vcard_string in zip(vcard_names, vcard_strings):
+            file_name = "".join(x for x in name if x.isalnum())
+            file_name = file_name.encode('utf-8')[:230].decode('utf-8', 'ignore')
+            file_path = os.path.join(path, f"{file_name}.vcf")
+            file_paths.append(file_path)
+
+            if not os.path.isfile(file_path):
+                with open(file_path, "w", encoding="utf-8") as f:
+                    f.write(vcard_string)
+
+        vcard_summary = "This media include the following vCard file(s):<br>" 
+        vcard_summary += " | ".join([f'<a href="{htmle(fp)}">{htmle(name)}</a>' for name, fp in zip(vcard_names, file_paths)])
+        message = data[content["ZCONTACTJID"]].messages[content["ZMESSAGE"]]
+        message.data = vcard_summary
         message.mime = "text/x-vcard"
         message.media = True
         message.meta = True
+        message.safe = True
         print(f"Processing vCards...({index + 1}/{total_row_number})", end="\r")

Whatsapp_Chat_Exporter/ios_media_handler.py

@@ -6,29 +6,38 @@ import os
 import time
 import getpass
 import threading
+from Whatsapp_Chat_Exporter.utility import WhatsAppIdentifier
 try:
     from iphone_backup_decrypt import EncryptedBackup, RelativePath
-    from iphone_backup_decrypt import FailedToDecryptError, Domain
+    from iphone_backup_decrypt import FailedToDecryptError
 except ModuleNotFoundError:
     support_encrypted = False
 else:
     support_encrypted = True
 
 
-def extract_encrypted(base_dir, password):
+def extract_encrypted(base_dir, password, identifiers, bplist_reader=None):
     backup = EncryptedBackup(backup_directory=base_dir, passphrase=password, cleanup=False, check_same_thread=False)
-    print("Decrypting WhatsApp database...")
+    print("Decrypting WhatsApp database...", end="")
     try:
-        backup.extract_file(relative_path=RelativePath.WHATSAPP_MESSAGES,
-                        output_filename="7c7fba66680ef796b916b067077cc246adacf01d")
-        backup.extract_file(relative_path=RelativePath.WHATSAPP_CONTACTS,
-                            output_filename="b8548dc30aa1030df0ce18ef08b882cf7ab5212f")
+        backup.extract_file(
+            relative_path=RelativePath.WHATSAPP_MESSAGES,
+            domain=identifiers.DOMAIN,
+            output_filename=identifiers.MESSAGE
+        )
+        backup.extract_file(
+            relative_path=RelativePath.WHATSAPP_CONTACTS,
+            domain=identifiers.DOMAIN,
+            output_filename=identifiers.CONTACT
+        )
     except FailedToDecryptError:
         print("Failed to decrypt backup: incorrect password?")
         exit()
+    else:
+        print("Done")
     extract_thread = threading.Thread(
         target=backup.extract_files_by_domain,
-        args=(Domain.WHATSAPP, Domain.WHATSAPP)
+        args=(identifiers.DOMAIN, identifiers.DOMAIN, bplist_reader)
     )
     extract_thread.daemon = True
     extract_thread.start()
@@ -61,7 +70,10 @@ def is_encrypted(base_dir):
             return False
 
 
-def extract_media(base_dir):
+def extract_media(base_dir, identifiers, preserve_timestamp=False):
+    if preserve_timestamp:
+        from Whatsapp_Chat_Exporter.bplist import BPListReader 
+        preserve_timestamp = BPListReader
     if is_encrypted(base_dir):
         if not support_encrypted:
             print("You don't have the dependencies to handle encrypted backup.")
@@ -70,21 +82,24 @@ def extract_media(base_dir):
             return False
         print("Encryption detected on the backup!")
         password = getpass.getpass("Enter the password for the backup:")
-        extract_encrypted(base_dir, password)
+        extract_encrypted(base_dir, password, identifiers, preserve_timestamp)
     else:
-        wts_db = os.path.join(base_dir, "7c/7c7fba66680ef796b916b067077cc246adacf01d")
-        contact_db = os.path.join(base_dir, "b8/b8548dc30aa1030df0ce18ef08b882cf7ab5212f")
+        wts_db = os.path.join(base_dir, identifiers.MESSAGE[:2], identifiers.MESSAGE)
+        contact_db = os.path.join(base_dir, identifiers.CONTACT[:2], identifiers.CONTACT)
         if not os.path.isfile(wts_db):
-            print("WhatsApp database not found.")
+            if identifiers is WhatsAppIdentifier:
+                print("WhatsApp database not found.")
+            else:
+                print("WhatsApp Business database not found.")
             exit()
         else:
-            shutil.copyfile(wts_db, "7c7fba66680ef796b916b067077cc246adacf01d")
+            shutil.copyfile(wts_db, identifiers.MESSAGE)
         if not os.path.isfile(contact_db):
             print("Contact database not found.")
             exit()
         else:
-            shutil.copyfile(contact_db, "b8548dc30aa1030df0ce18ef08b882cf7ab5212f")
-        _wts_id = "AppDomainGroup-group.net.whatsapp.WhatsApp.shared"
+            shutil.copyfile(contact_db, identifiers.CONTACT)
+        _wts_id = identifiers.DOMAIN
         with sqlite3.connect(os.path.join(base_dir, "Manifest.db")) as manifest:
             manifest.row_factory = sqlite3.Row
             c = manifest.cursor()
@@ -98,6 +113,7 @@ def extract_media(base_dir):
             c.execute(f"""SELECT fileID,
                                 relativePath,
                                 flags,
+                                file AS metadata,
                                 ROW_NUMBER() OVER(ORDER BY relativePath) AS _index
                         FROM Files
                         WHERE domain = '{_wts_id}'
@@ -120,6 +136,11 @@ def extract_media(base_dir):
                         pass
                 elif flags == 1:
                     shutil.copyfile(os.path.join(base_dir, folder, hashes), destination)
+                    if preserve_timestamp:
+                        metadata = BPListReader(row["metadata"]).parse()
+                        creation = metadata["$objects"][1]["Birth"]
+                        modification = metadata["$objects"][1]["LastModified"]
+                        os.utime(destination, (modification, modification))
                 if row["_index"] % 100 == 0:
                     print(f"Extracting WhatsApp files...({row['_index']}/{total_row_number})", end="\r")
                 row = c.fetchone()

Whatsapp_Chat_Exporter/utility.py

@@ -3,17 +3,21 @@ import json
 import os
 from bleach import clean as sanitize
 from markupsafe import Markup
+import unicodedata
+import re
 from datetime import datetime
 from enum import IntEnum
 from Whatsapp_Chat_Exporter.data_model import ChatStore
 try:
-    from enum import StrEnum
+    from enum import StrEnum, IntEnum
 except ImportError:
     # < Python 3.11
     from enum import Enum
     class StrEnum(str, Enum):
         pass
 
+    class IntEnum(int, Enum):
+        pass
 
 MAX_SIZE = 4 * 1024 * 1024  # Default 4MB
 ROW_SIZE = 0x3D0
@@ -154,6 +158,16 @@ def get_file_name(contact: str, chat: ChatStore):
     return "".join(x for x in file_name if x.isalnum() or x in "- "), name
 
 
+def get_chat_condition(filter, include, column):
+    if filter is not None:
+        if include:
+            return f'''AND ({' OR '.join(f"{column} LIKE '%{chat}%'" for chat in filter)})'''
+        else:
+            return f'''AND ({' AND '.join(f"{column} NOT LIKE '%{chat}%'" for chat in filter)})'''
+    else:
+        return ""
+
+
 # Android Specific
 CRYPT14_OFFSETS = (
     {"iv": 67, "db": 191},
@@ -170,6 +184,11 @@ class Crypt(IntEnum):
     CRYPT12 = 12
 
 
+class DbType(StrEnum):
+    MESSAGE = "message"
+    CONTACT = "contact"
+
+
 def brute_force_offset(max_iv=200, max_db=200):
     for iv in range(0, max_iv):
         for db in range(0, max_db):
@@ -290,3 +309,38 @@ def setup_template(template, no_avatar):
 
 # iOS Specific
 APPLE_TIME = datetime.timestamp(datetime(2001, 1, 1))
+
+
+def slugify(value, allow_unicode=False):
+    """
+    Taken from https://github.com/django/django/blob/master/django/utils/text.py
+    Convert to ASCII if 'allow_unicode' is False. Convert spaces or repeated
+    dashes to single dashes. Remove characters that aren't alphanumerics,
+    underscores, or hyphens. Convert to lowercase. Also strip leading and
+    trailing whitespace, dashes, and underscores.
+    """
+    value = str(value)
+    if allow_unicode:
+        value = unicodedata.normalize('NFKC', value)
+    else:
+        value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore').decode('ascii')
+    value = re.sub(r'[^\w\s-]', '', value.lower())
+    return re.sub(r'[-\s]+', '-', value).strip('-_')
+
+
+class WhatsAppIdentifier(StrEnum):
+    MESSAGE = "7c7fba66680ef796b916b067077cc246adacf01d"
+    CONTACT = "b8548dc30aa1030df0ce18ef08b882cf7ab5212f"
+    DOMAIN = "AppDomainGroup-group.net.whatsapp.WhatsApp.shared"
+
+
+class WhatsAppBusinessIdentifier(StrEnum):
+    MESSAGE = "724bd3b98b18518b455a87c1f3ac3a0d189c4466"
+    CONTACT = "d7246a707f51ddf8b17ee2dddabd9e0a4da5c552"
+    DOMAIN = "AppDomainGroup-group.net.whatsapp.WhatsAppSMB.shared"
+
+class JidType(IntEnum):
+    PM = 0
+    GROUP = 1
+    SYSTEM_BROADCAST = 5
+    STATUS = 11

Whatsapp_Chat_Exporter/whatsapp.html

@@ -139,7 +139,7 @@
 								</div>
 							{% else %}
 								{% if msg.media == false %}
-								{{ msg.data | sanitize_except() }}
+								{{ msg.data | sanitize_except() | urlize(none, true, '_blank') }}
 								{% else %}
 									{% if "image/" in msg.mime %}
 									<a href="{{ msg.data }}">
@@ -162,7 +162,7 @@
 									{% endif %}
 									{% if msg.caption is not none %}
 									<br>
-									{{ msg.caption }}
+									{{ msg.caption | urlize(none, true, '_blank') }}
 									{% endif %}
 								{% endif %}
 								{% endif %}
@@ -223,7 +223,7 @@
 								</div>
 							{% else %}
 								{% if msg.media == false %}
-								{{ msg.data | sanitize_except() }}
+								{{ msg.data | sanitize_except() | urlize(none, true, '_blank') }}
 								{% else %}
 									{% if "image/" in msg.mime %}
 									<a href="{{ msg.data }}">
@@ -246,7 +246,7 @@
 									{% endif %}
 									{% if msg.caption is not none %}
 									<br>
-									{{ msg.caption }}
+									{{ msg.caption | urlize(none, true, '_blank') }}
 									{% endif %}
 								{% endif %}
 							{% endif %}

docs.html

@@ -0,0 +1,20 @@
+<!DOCTYPE html>
+<html>
+    <head>
+        <meta http-equiv="refresh" content="0; url='https://github.com/KnugiHK/WhatsApp-Chat-Exporter/wiki'" />
+        <script type="text/javascript">
+            destination = {
+                "filter": "Filter",
+                "date": "Filters#date-filters",
+                "chat": "Filters#chat-filter",
+                "osl": "Open-Source-Licenses"
+                null: ""
+            };
+            const dest = new URLSearchParams(window.location.search).get('dest');
+            window.location.href = `https://github.com/KnugiHK/WhatsApp-Chat-Exporter/wiki/${destination[dest]}`;
+        </script>
+    </head>
+    <body>
+        <p>If the redirection doesn't work, you can find the documentation at <a href="https://github.com/KnugiHK/WhatsApp-Chat-Exporter/wiki">https://github.com/KnugiHK/WhatsApp-Chat-Exporter/wiki</a>.</p>
+    </body>
+</html>

scripts/bruteforce_crypt15.py

@@ -0,0 +1,48 @@
+import hmac
+import javaobj
+import zlib
+from Crypto.Cipher import AES
+from hashlib import sha256
+
+
+def _generate_hmac_of_hmac(key_stream):
+    key = hmac.new(
+        hmac.new(
+            b'\x00' * 32,
+            key_stream,
+            sha256
+        ).digest(),
+        b"backup encryption\x01",
+        sha256
+    )
+    return key.digest(), key_stream
+
+
+def _extract_encrypted_key(keyfile):
+    key_stream = b""
+    for byte in javaobj.loads(keyfile):
+        key_stream += byte.to_bytes(1, "big", signed=True)
+
+    return _generate_hmac_of_hmac(key_stream)
+
+
+key = open("encrypted_backup.key", "rb").read()
+database = open("wa.db.crypt15", "rb").read()
+main_key, hex_key = _extract_encrypted_key(key)
+for i in range(100):
+    iv = database[i:i+16]
+    for j in range(100):
+        cipher = AES.new(main_key, AES.MODE_GCM, iv)
+        db_ciphertext = database[j:]
+        db_compressed = cipher.decrypt(db_ciphertext)
+        try:
+            db = zlib.decompress(db_compressed)
+        except zlib.error:
+            ...
+        else:
+            if db[0:6] == b"SQLite":
+                print(f"Found!\nIV: {i}\nOffset: {j}")
+                print(db_compressed[:10])
+                exit()
+
+print("Not found! Try to increase maximum search.")