WIP: Extract Android backups, yield devices instead of just echoing #80

rytilahti · 2017-09-17T19:29:17Z

Instead of requiring using Android backup extractor tool, this PR makes it possible to pass an Android backup (and a password if used) for the token extraction.

This moves the functionality towards automatic config generation out of backups.
The extraction depends on code which is not available in pypi, and requires some
changes to be useful for us: bluec0re/android-backup-tools#1

This is a WIP PR, let's hope the author of android-backup-tools agrees with the changes & inclusion to the pypi.

To test it out:

pip install git+https://github.com/rytilahti/android-backup-tools.git@sanitize_for_lib_use
pip install git+https://github.com/rytilahti/python-mirobo.git@extract_android_backups
miio-extract-tokens --password 1234 backup.ab

houndci-bot · 2017-09-17T19:29:28Z

mirobo/extract_tokens.py

+
+    for dev in devices:
+        if dev.ip or dump_all:
+            click.echo("%s\n\tModel: %s\n\tIP address: %s\n\tToken: %s\n\tMAC: %s" % (dev.name, dev.model, dev.ip, dev.token, dev.mac))


line too long (135 > 79 characters)

houndci-bot · 2017-09-17T19:29:29Z

mirobo/extract_tokens.py

+            try:
+                db = tar.extractfile(DBFILE)
+            except KeyError as ex:
+                click.echo("Unable to extract the device database file %s: %s" % (DBFILE, ex))


line too long (94 > 79 characters)

houndci-bot · 2017-09-17T19:29:29Z

mirobo/extract_tokens.py

            else:
                click.echo("Error, unknown database type!")


 @click.command()
 @click.argument('backup')
 @click.option('--write-to-disk', type=click.File('wb'), help='writes sqlite3 db to a file for debugging')
+@click.option('--password', type=str, help='password if the android database is encrypted')


line too long (91 > 79 characters)

houndci-bot · 2017-09-17T19:29:29Z

mirobo/extract_tokens.py

-        self.db.seek(0)  # go to the beginning
-        click.echo("Saving db to %s" % fp)
-        fp.write(self.db.read())
+            config = DeviceConfig(name=name, ip=ip, mac=mac, model=model, token=token)


line too long (86 > 79 characters)

houndci-bot · 2017-09-17T19:29:29Z

mirobo/extract_tokens.py

-            if ip or self.dump_all:
-                click.echo("%s\n\tModel: %s\n\tIP address: %s\n\tToken: %s\n\tMAC: %s" % (name, model, ip, token, mac))
+
+            config = DeviceConfig(name=name, mac=mac, ip=ip, model=model, token=token)


line too long (86 > 79 characters)

houndci-bot · 2017-09-17T19:29:29Z

mirobo/extract_tokens.py

@@ -4,11 +4,19 @@
 import sqlite3
 from Crypto.Cipher import AES
 from pprint import pformat as pf
+import attr
+
+@attr.s


expected 2 blank lines, found 1

This is one of the lint errors I would like to see fixed.

coveralls · 2017-09-17T19:30:41Z

Coverage decreased (-0.2%) to 32.582% when pulling 0777489 on extract_android_backups into e8ed75a on master.

syssi · 2017-09-19T06:12:19Z

mirobo/extract_tokens.py

@@ -4,11 +4,19 @@
 import sqlite3
 from Crypto.Cipher import AES
 from pprint import pformat as pf
+import attr
+
+@attr.s


This is one of the lint errors I would like to see fixed.

syssi · 2017-09-19T06:13:21Z

mirobo/extract_tokens.py


    def read_tokens(self, db):
        self.db = db
+        print("reading database from %s" % db)


print vs. click.echo

Changed to use logger.

syssi · 2017-09-19T06:17:19Z

mirobo/extract_tokens.py

    """Reads device information out from an sqlite3 DB.
-     If the given file is a .tar file, the file will be extracted
-     and the database automatically located (out of Android backups).
+     If the given file is an .ab file, it will be extracted


If the given file is an Android backup (*.ab) the database will be extracted automatically.

Thanks, I also added a note about iOS DBs, but I hope this extraction part will be just a temporary solution and that the DB code can be used for configuration handling for the unified cli tool.

houndci-bot · 2017-09-20T17:41:19Z

mirobo/extract_tokens.py

+            try:
+                db = tar.extractfile(DBFILE)
+            except KeyError as ex:
+                click.echo("Unable to extract the database file %s: %s" % (DBFILE, ex))


line too long (87 > 79 characters)

coveralls · 2017-09-20T17:42:55Z

Coverage decreased (-0.2%) to 32.549% when pulling c185c15 on extract_android_backups into e8ed75a on master.

rytilahti · 2017-09-22T20:55:58Z

I think this is safe to be merged. Hound being hound and all with its too narrow line length limits, still waiting to see if the android-backup-tools author will give it a go, or if there is a need to find another solution/make a fork :-)

houndci-bot · 2017-10-01T17:06:00Z

mirobo/android_backup.py

+
+        if self.encryption == EncryptionType.AES256:
+            if password is None:
+                raise Exception("Password need to be provided to extract encrypted archives")


line too long (93 > 79 characters)

houndci-bot · 2017-10-01T17:06:01Z

mirobo/android_backup.py

+                binascii.b2a_hex(master_salt).upper() + b"\n" + \
+                str(rounds).encode() + b"\n" + \
+                binascii.b2a_hex(user_iv).upper() + b"\n" + \
+                binascii.b2a_hex(master_enc).upper() + b"\n" + enc


continuation line over-indented for visual indent

houndci-bot · 2017-10-01T17:06:01Z

mirobo/android_backup.py

+        enc = binascii.b2a_hex(user_salt).upper() + b"\n" + \
+                binascii.b2a_hex(master_salt).upper() + b"\n" + \
+                str(rounds).encode() + b"\n" + \
+                binascii.b2a_hex(user_iv).upper() + b"\n" + \


continuation line over-indented for visual indent

houndci-bot · 2017-10-01T17:06:01Z

mirobo/android_backup.py

+
+        enc = binascii.b2a_hex(user_salt).upper() + b"\n" + \
+                binascii.b2a_hex(master_salt).upper() + b"\n" + \
+                str(rounds).encode() + b"\n" + \


continuation line over-indented for visual indent

houndci-bot · 2017-10-01T17:06:01Z

mirobo/android_backup.py

+        master_enc = cipher.encrypt(master_dec)
+
+        enc = binascii.b2a_hex(user_salt).upper() + b"\n" + \
+                binascii.b2a_hex(master_salt).upper() + b"\n" + \


continuation line over-indented for visual indent

houndci-bot · 2017-10-01T17:06:01Z

mirobo/android_backup.py

+        user_key = PBKDF2(password,
+                          user_salt, dkLen=256//8, count=rounds)
+
+        master_dec = b"\x10" + master_iv + b"\x20" + master_key + b"\x20" + master_ck


line too long (85 > 79 characters)

This moves the functionality towards automatic config generation out of backups. The extraction depends on code which is not available in pypi, and requires some changes to be useful for us: bluec0re/android-backup-tools#1

…ll changes based on code review

Forked and modified from https://github.com/bluec0re/android-backup-tools

rytilahti · 2017-10-01T17:07:49Z

Forked the modified version from the android-backup-tools, ignoring the style problems for now.

coveralls · 2017-10-01T17:08:14Z

Coverage decreased (-2.6%) to 30.198% when pulling 341e709 on extract_android_backups into a6a8ecd on master.

coveralls · 2017-10-01T17:08:40Z

Coverage decreased (-2.6%) to 30.198% when pulling 341e709 on extract_android_backups into a6a8ecd on master.

houndci-bot reviewed Sep 17, 2017

View reviewed changes

syssi reviewed Sep 19, 2017

View reviewed changes

houndci-bot reviewed Sep 20, 2017

View reviewed changes

houndci-bot reviewed Oct 1, 2017

View reviewed changes

rytilahti added 3 commits October 1, 2017 19:06

Extract Android backups, yield devices instead of just echoing

922c3b0

This moves the functionality towards automatic config generation out of backups. The extraction depends on code which is not available in pypi, and requires some changes to be useful for us: bluec0re/android-backup-tools#1

use logging for the database class, fix linting mistakes and some sma…

d67c2d9

…ll changes based on code review

Add a fork of python-backup-tools for backup extraction

341e709

Forked and modified from https://github.com/bluec0re/android-backup-tools

rytilahti force-pushed the extract_android_backups branch from 9574fcf to 341e709 Compare October 1, 2017 17:06

rytilahti merged commit e6e72e0 into master Oct 1, 2017

rytilahti deleted the extract_android_backups branch October 1, 2017 17:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Extract Android backups, yield devices instead of just echoing #80

WIP: Extract Android backups, yield devices instead of just echoing #80

rytilahti commented Sep 17, 2017 •

edited

Loading

houndci-bot Sep 17, 2017

houndci-bot Sep 17, 2017

houndci-bot Sep 17, 2017

houndci-bot Sep 17, 2017

houndci-bot Sep 17, 2017

houndci-bot Sep 17, 2017

syssi Sep 19, 2017

coveralls commented Sep 17, 2017 •

edited

Loading

syssi Sep 19, 2017

syssi Sep 19, 2017

rytilahti Sep 20, 2017

syssi Sep 19, 2017

rytilahti Sep 20, 2017

houndci-bot Sep 20, 2017

coveralls commented Sep 20, 2017 •

edited

Loading

rytilahti commented Sep 22, 2017 •

edited

Loading

houndci-bot Oct 1, 2017

houndci-bot Oct 1, 2017

houndci-bot Oct 1, 2017

houndci-bot Oct 1, 2017

houndci-bot Oct 1, 2017

houndci-bot Oct 1, 2017

rytilahti commented Oct 1, 2017

coveralls commented Oct 1, 2017 •

edited

Loading

coveralls commented Oct 1, 2017 •

edited

Loading

WIP: Extract Android backups, yield devices instead of just echoing #80

WIP: Extract Android backups, yield devices instead of just echoing #80

Conversation

rytilahti commented Sep 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Sep 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Sep 20, 2017 • edited Loading

rytilahti commented Sep 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rytilahti commented Oct 1, 2017

coveralls commented Oct 1, 2017 • edited Loading

coveralls commented Oct 1, 2017 • edited Loading

rytilahti commented Sep 17, 2017 •

edited

Loading

coveralls commented Sep 17, 2017 •

edited

Loading

coveralls commented Sep 20, 2017 •

edited

Loading

rytilahti commented Sep 22, 2017 •

edited

Loading

coveralls commented Oct 1, 2017 •

edited

Loading

coveralls commented Oct 1, 2017 •

edited

Loading