Merge branch 'dev'

Fixed bug where blank VCard media_name would crash the program.

.github/workflows/python-publish.yml

@@ -0,0 +1,36 @@
+# This workflow will upload a Python Package using Twine when a release is created
+# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+name: Upload Python Package
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  deploy:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.x'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install build
+    - name: Build package
+      run: python -m build
+    - name: Publish package
+      uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
+      with:
+        user: __token__
+        password: ${{ secrets.PYPI_API_TOKEN }}

README.md

@@ -7,6 +7,8 @@ A customizable Android and iPhone Whatsapp database parser that will give you th
 **If you plan to uninstall WhatsApp or delete your WhatsApp account, please make a backup of your WhatsApp database. You may want to use this exporter again on the same database in the future as the exporter develops**
 
 # Usage
+**Usage in README may be removed in the future. Check the usage in [Wiki](https://github.com/KnugiHK/Whatsapp-Chat-Exporter/wiki)**
+
 **If you want to use the old release (< 0.5) of the exporter, please follow the [old usage guide](https://github.com/KnugiHK/Whatsapp-Chat-Exporter/blob/main/old_README.md#usage)**
 
 First, install the exporter by:
@@ -23,7 +25,7 @@ cd working_wts
 ### Unencrypted WhatsApp database
 Extract the WhatsApp database with whatever means, one possible means is to use the [WhatsApp-Key-DB-Extractor](https://github.com/KnugiHK/WhatsApp-Key-DB-Extractor)
 
-After you obtain your WhatsApp databse, copy the WhatsApp database and media folder to the working directory. The database is called msgstore.db. If you also want the name of your contacts, get the contact database, which is called wa.db. And copy the WhatsApp (Media) directory from your phone directly.
+After you obtain your WhatsApp database, copy the WhatsApp database and media folder to the working directory. The database is called msgstore.db. If you also want the name of your contacts, get the contact database, which is called wa.db. And copy the WhatsApp (Media) directory from your phone directly.
 
 And now, you should have something like this in the working directory.
 
@@ -37,8 +39,10 @@ wtsexporter -a
 ### Encrypted Android WhatsApp Backup
 In order to support the decryption, install pycryptodome if it is not installed
 ```sh
-pip install pycryptodome
+pip install pycryptodome # Or 
+pip install whatsapp-chat-exporter["android_backup"] # install along with this software
 ```
+#### Crypt12 or Crypt14
 Place the decryption key file (key) and the encrypted WhatsApp Backup (msgstore.db.crypt14) in the working directory. If you also want the name of your contacts, get the contact database, which is called wa.db. And copy the WhatsApp (Media) directory from your phone directly.
 
 And now, you should have something like this in the working directory.
@@ -50,6 +54,28 @@ Simply invoke the following command from shell.
 wtsexporter -a -k key -b msgstore.db.crypt14
 ```
 
+#### Crypt15 (End-to-End Encrypted Backup)
+To support Crypt15 backup, install javaobj-py3 if it is not installed
+```sh
+pip install javaobj-py3 # Or 
+pip install whatsapp-chat-exporter["crypt15"] # install along with this software
+```
+Place the encrypted WhatsApp Backup (msgstore.db.crypt15) in the working directory. If you also want the name of your contacts, get the contact database, which is called wa.db. And copy the WhatsApp (Media) directory from your phone directly.  
+If you do not have the 32 bytes hex key (64 hexdigits), place the decryption key file (encrypted_backup.key) extracted from Android. If you gave the 32 bytes hex key, simply put the key in the shell.
+
+Now, you should have something like this in the working directory (if you do not have 32 bytes hex key).
+
+![Android folder structure with WhatsApp Crypt15 Backup](imgs/android_structure_backup_crypt15.png)
+##### Extracting
+If you do not have 32 bytes hex key but have the key file available, simply invoke the following command from shell.
+```sh
+wtsexporter -a -k encrypted_backup.key -b msgstore.db.crypt15
+```
+If you have the 32 bytes hex key, simply put the hex key in the -k option and invoke the command from shell like this:
+```sh
+wtsexporter -a -k 432435053b5204b08e5c3823423399aa30ff061435ab89bc4e6713969cdaa5a8 -b msgstore.db.crypt15
+```
+
 ## Working with iPhone
 Do an iPhone Backup with iTunes first.
 ### Encrypted iPhone Backup

Whatsapp_Chat_Exporter/__init__.py

@@ -1 +1 @@
-__version__ = "0.7.0"
+__version__ = "0.8.2"

Whatsapp_Chat_Exporter/__main__.py

@@ -1,11 +1,14 @@
 from .__init__ import __version__
 from Whatsapp_Chat_Exporter import extract, extract_iphone
 from Whatsapp_Chat_Exporter import extract_iphone_media
+from Whatsapp_Chat_Exporter.extract import Crypt
 from optparse import OptionParser
 import os
 import sqlite3
 import shutil
 import json
+import string
+from sys import exit
 
 
 def main():
@@ -75,14 +78,21 @@ def main():
         dest="template",
         default=None,
         help="Path to custom HTML template")
+    parser.add_option(
+        "-e",
+        "--embedded",
+        dest="embedded",
+        default=False,
+        action='store_true',
+        help="Embed media into HTML file")
     (options, args) = parser.parse_args()
 
     if options.android and options.iphone:
         print("You must define only one device type.")
-        exit()
+        exit(1)
     if not options.android and not options.iphone:
         print("You must define the device type.")
-        exit()
+        exit(1)
     data = {}
 
     if options.android:
@@ -98,15 +108,32 @@ def main():
         if options.key is not None:
             if options.backup is None:
                 print("You must specify the backup file with -b")
-                return False
+                exit(1)
             print("Decryption key specified, decrypting WhatsApp backup...")
-            key = open(options.key, "rb").read()
+            if "crypt12" in options.backup:
+                crypt = Crypt.CRYPT12
+            elif "crypt14" in options.backup:
+                crypt = Crypt.CRYPT14
+            elif "crypt15" in options.backup:
+                crypt = Crypt.CRYPT15
+            if os.path.isfile(options.key):
+                key = open(options.key, "rb")
+            elif all(char in string.hexdigits for char in options.key):
+                key = bytes.fromhex(options.key)
             db = open(options.backup, "rb").read()
-            is_crypt14 = False if "crypt12" in options.backup else True
-            if not extract.decrypt_backup(db, key, msg_db, is_crypt14):
-                print("Dependencies of decrypt_backup are not "
-                      "present. For details, see README.md")
-                return False
+            error = extract.decrypt_backup(db, key, msg_db, crypt)
+            if error != 0:
+                if error == 1:
+                    print("Dependencies of decrypt_backup and/or extract_encrypted_key"
+                          " are not present. For details, see README.md.")
+                    exit(3)
+                elif error == 2:
+                    print("Failed when decompressing the decrypted backup. "
+                          "Possibly incorrect offsets used in decryption.")
+                    exit(4)
+                else:
+                    print("Unknown error occurred.")
+                    exit(5)
         if options.wa is None:
             contact_db = "wa.db"
         else:
@@ -147,10 +174,22 @@ def main():
             messages(db, data)
             media(db, data, options.media)
             vcard(db, data)
-        create_html(data, options.output, options.template)
+        create_html(data, options.output, options.template, options.embedded)
+    else:
+        print(
+            "The message database does not exist. You may specify the path "
+            "to database file with option -d or check your provided path.",
+            end="\r"
+        )
+        exit(2)
 
-    if not os.path.isdir(f"{options.output}/{options.media}"):
-        shutil.move(options.media, f"{options.output}/")
+    if os.path.isdir(options.media) and \
+            not os.path.isdir(f"{options.output}/{options.media}"):
+        try:
+            shutil.move(options.media, f"{options.output}/")
+        except PermissionError:
+            print("Cannot remove original WhatsApp directory. "
+                  "Perhaps the directory is opened?")
 
     if options.json:
         with open("result.json", "w") as f:

Whatsapp_Chat_Exporter/extract.py

@@ -8,11 +8,16 @@ import requests
 import shutil
 import re
 import pkgutil
+import io
+import hmac
 from pathlib import Path
 from bleach import clean as sanitize
 from markupsafe import Markup
 from datetime import datetime
+from enum import Enum
 from mimetypes import MimeTypes
+from hashlib import sha256
+
 try:
     import zlib
     from Crypto.Cipher import AES
@@ -20,7 +25,12 @@ except ModuleNotFoundError:
     support_backup = False
 else:
     support_backup = True
-
+try:
+    import javaobj
+except ModuleNotFoundError:
+    support_crypt15 = False
+else:
+    support_crypt15 = True
 
 def sanitize_except(html):
     return Markup(sanitize(html, tags=["br"]))
@@ -34,38 +44,135 @@ def determine_day(last, current):
     else:
         return current
 
+CRYPT14_OFFSETS = [
+    {"iv": 67, "db": 191},
+    {"iv": 67, "db": 190},
+    {"iv": 66, "db": 99}
+]
 
-def decrypt_backup(database, key, output, crypt14=True):
+
+class Crypt(Enum):
+    CRYPT15 = 15
+    CRYPT14 = 14
+    CRYPT12 = 12
+
+
+def brute_force_offset():
+    for iv in range(0, 200):
+        for db in range(0, 200):
+            yield iv, iv + 16, db
+
+
+def _generate_hmac_of_hmac(key_stream):
+    key = hmac.new(
+        hmac.new(
+            b'\x00' * 32,
+            key_stream,
+            sha256
+        ).digest(),
+        b"backup encryption\x01",
+        sha256
+    )
+    return key.digest()
+
+
+def _extract_encrypted_key(keyfile):
+    key_stream = b""
+    for byte in javaobj.loads(keyfile):
+        key_stream += byte.to_bytes(1, "big", signed=True)
+
+    return _generate_hmac_of_hmac(key_stream)
+    
+
+def decrypt_backup(database, key, output, crypt=Crypt.CRYPT14):
     if not support_backup:
-        return False
-    if len(key) != 158:
+        return 1
+    if isinstance(key, io.IOBase):
+        key = key.read()
+        if crypt is not Crypt.CRYPT15:
+            t1 = key[30:62]
+    if crypt is not Crypt.CRYPT15 and len(key) != 158:
         raise ValueError("The key file must be 158 bytes")
-    t1 = key[30:62]
-    if crypt14:
+    if crypt == Crypt.CRYPT14:
         if len(database) < 191:
             raise ValueError("The crypt14 file must be at least 191 bytes")
+        current_try = 0
+        offsets = CRYPT14_OFFSETS[current_try]
         t2 = database[15:47]
-        iv = database[67:83]
-        db_ciphertext = database[191:]
-    else:
+        iv = database[offsets["iv"]:offsets["iv"] + 16]
+        db_ciphertext = database[offsets["db"]:]
+    elif crypt == Crypt.CRYPT12:
         if len(database) < 67:
             raise ValueError("The crypt12 file must be at least 67 bytes")
         t2 = database[3:35]
         iv = database[51:67]
         db_ciphertext = database[67:-20]
+    elif crypt == Crypt.CRYPT15:
+        if not support_crypt15:
+            return 1
+        if len(database) < 131:
+            raise ValueError("The crypt15 file must be at least 131 bytes")
+        t1 = t2 = None
+        iv = database[8:24]
+        db_ciphertext = database[131:]
+
     if t1 != t2:
         raise ValueError("The signature of key file and backup file mismatch")
 
-    main_key = key[126:]
-    cipher = AES.new(main_key, AES.MODE_GCM, iv)
-    db_compressed = cipher.decrypt(db_ciphertext)
-    db = zlib.decompress(db_compressed)
-    if db[0:6].upper() == b"SQLITE":
-        with open(output, "wb") as f:
-            f.write(db)
-        return True
+    if crypt == Crypt.CRYPT15:
+        if len(key) == 32:
+            main_key = _generate_hmac_of_hmac(key)
+        else:
+            main_key = _extract_encrypted_key(key)
     else:
-        raise ValueError("The plaintext is not a SQLite database. Did you use the key to encrypt something...")
+        main_key = key[126:]
+    decompressed = False
+    while not decompressed:
+        cipher = AES.new(main_key, AES.MODE_GCM, iv)
+        db_compressed = cipher.decrypt(db_ciphertext)
+        try:
+            db = zlib.decompress(db_compressed)
+        except zlib.error:
+            if crypt == Crypt.CRYPT14:
+                current_try += 1
+                if current_try < len(CRYPT14_OFFSETS):
+                    offsets = CRYPT14_OFFSETS[current_try]
+                    iv = database[offsets["iv"]:offsets["iv"] + 16]
+                    db_ciphertext = database[offsets["db"]:]
+                    continue
+                else:
+                    print("Common offsets are not applicable to "
+                          "your backup. Trying to brute force it...")
+                    for start_iv, end_iv, start_db in brute_force_offset():
+                        iv = database[start_iv:end_iv]
+                        db_ciphertext = database[start_db:]
+                        cipher = AES.new(main_key, AES.MODE_GCM, iv)
+                        db_compressed = cipher.decrypt(db_ciphertext)
+                        try:
+                            db = zlib.decompress(db_compressed)
+                        except zlib.error:
+                            continue
+                        else:
+                            decompressed = True
+                            print(
+                                f"The offsets of your IV and database are {start_iv} and "
+                                f"{start_db}, respectively. To include your offsets in the "
+                                "program, please report it by creating an issue on GitHub: "
+                                "https://github.com/KnugiHK/Whatsapp-Chat-Exporter/issues/new"
+                            )
+                            break
+                    if not decompressed:
+                        return 2
+            else:
+                return 3
+        else:
+            decompressed = True
+        if db[0:6].upper() == b"SQLITE":
+            with open(output, "wb") as f:
+                f.write(db)
+            return 0
+        else:
+            raise ValueError("The plaintext is not a SQLite database. Did you use the key to encrypt something...")
 
 
 def contacts(db, data):
@@ -108,7 +215,8 @@ def messages(db, data):
                         messages.media_caption
                  FROM messages
                     LEFT JOIN messages_quotes
-                        ON messages.quoted_row_id = messages_quotes._id;""")
+                        ON messages.quoted_row_id = messages_quotes._id
+                 WHERE messages.key_remote_jid <> '-1';""")
     i = 0
     content = c.fetchone()
     while content is not None:
@@ -307,12 +415,13 @@ def vcard(db, data):
     if not os.path.isdir(base):
         Path(base).mkdir(parents=True, exist_ok=True)
     for index, row in enumerate(rows):
-        file_name = "".join(x for x in row[3] if x.isalnum())
+        media_name = row[3] if row[3] else ""
+        file_name = "".join(x for x in media_name if x.isalnum())
         file_path = f"{base}/{file_name}.vcf"
         if not os.path.isfile(file_path):
             with open(file_path, "w", encoding="utf-8") as f:
                 f.write(row[2])
-        data[row[1]]["messages"][row[0]]["data"] = row[3] + \
+        data[row[1]]["messages"][row[0]]["data"] = media_name + \
             "The vCard file cannot be displayed here, " \
             f"however it should be located at {file_path}"
         data[row[1]]["messages"][row[0]]["mime"] = "text/x-vcard"
@@ -320,7 +429,7 @@ def vcard(db, data):
         print(f"Gathering vCards...({index + 1}/{total_row_number})", end="\r")
 
 
-def create_html(data, output_folder, template=None):
+def create_html(data, output_folder, template=None, embedded=False):
     if template is None:
         template_dir = os.path.dirname(__file__)
         template_file = "whatsapp.html"

Whatsapp_Chat_Exporter/extract_iphone.py

@@ -230,7 +230,7 @@ def vcard(db, data):
         print(f"Gathering vCards...({index + 1}/{total_row_number})", end="\r")
 
 
-def create_html(data, output_folder, template=None):
+def create_html(data, output_folder, template=None, embedded=False):
     if template is None:
         template_dir = os.path.dirname(__file__)
         template_file = "whatsapp.html"

_config.yml

@@ -1 +0,0 @@
-theme: jekyll-theme-cayman

setup.py

@@ -41,7 +41,10 @@ setuptools.setup(
        'bleach'
     ],
     extras_require={
-        'android_backup':  ["pycryptodome"]
+        'android_backup':  ["pycryptodome", "javaobj-py3"],
+        'crypt12':  ["pycryptodome"],
+        'crypt12':  ["pycryptodome"],
+        'crypt15':  ["pycryptodome", "javaobj-py3"]
     },
     entry_points={
         "console_scripts": [