Android Email Extraction to .eml

sometimes the android ecosystem is a little lacking with tool support; for instance i needed to extract a set of sent items from a pop3 mailbox – the stock mail client only allows you to perform 3 actions: delete, mark as unread or favourite.

armed with the android sdk, some sql queries and a groovy script we’ll see how it’s possible to recover email to rfc822 .eml files.

email storage

the stock com.android.email client stores message headers and their bodies in two sqlite databases and the attachmentprovider stores attachments on disk.

the source code is available from https://android.googlesource.com/platform/packages/apps/email – in this case it was for a samsung phone running 4.1.2 so i checked out the jb-mr0-release branch to inspect the code when required.

backup with adb

adb is the android debug bridge , a debugging tool that is part of the android sdk platform tools . executing ‘ adb usb ‘ (re)starts the adbd daemon listening to usb connections.

the next step is to select developer options from the system section of the settings list (figure 1) and then enable usb debugging on the device (figure 2).

figure 1 – system settings

figure 2 – developer options

then connect to the device using a usb cable and execute ‘ adb backup -f mybackup.ab -all ‘ (you can also be more selective with the package you want, e.g. ‘ adb backup -f mybackup.ab com.android.email ‘). adb will prompt you to unlock the phone and permit the backup to proceed.

inflating the backup

an android backup is a zlib deflated tar file – thanks to http://nelenkov.blogspot.jp/2012/06/unpacking-android-backups.html for the hint that it can be reinflated with the following command:
dd if=mybackup.ab bs=24 skip=1|openssl zlib -d > mybackup.tar

note that the install of openssl i had on osx wasn’t compiled with zlib support, so i ran this through an ubuntu vagrant vm to inflate the tar file.

examining the backup

unpacking the tar file will give you a folder structure as shown in figure 3; in this case the sqlite databases are in db and photo attachments taken with the phone are in f .

figure 3 – expanded tar

exploring the databases

sqlitebrowser is a good tool for browsing sqlite database and features in the following screenshots.

the current script restricts the messages processed to a single mailbox for one account. to determine the mailboxkey requires locating the target account from the account table (figure 4) and then isolating the desired mailbox for that account within the mailbox table (figure 5).

figure 4 – account table structure

figure 5 – mailbox table structure

you can browse the data or execute the following query to get a list of mailboxes and their corresponding accounts:

select a.displayname as accountname, m._id as mailboxkey, m.displayname as mailboxname
from account a, mailbox m
where m.accountkey = a._id

if you don’t want to limit the extraction to a single mailbox or account then you can remove the where clause from the first select query.

email extraction

being comfortable with the rfc822 specs & javamail (i built an imap extension for alfresco in 2006), i decided it would be easier to reconstruct a mimemessage using sql & groovy than attempt to adapt the android source code to run off the device or build a custom app to run in the emulator.

shortcomings

as a first cut that was good enough for my purposes, there are a few deficiencies:
1. it is set to default the sender field rather than converting the value of message.fromlist and using the addfrom method
2. addresses only use the address rather than the label
3. the body handling only uses plain text rather than alternative multiparts
4. body.textreply is separated from the body.textcontent with a separator line of 25 dashes; it does not attempt to reconstruct the header information of the preceding message in the thread
5. the script does not handle attachments – this was a conscious decision as whilst the camera photos were in com.android.email/f , the other ‘raw’ attachments were not in com.android.email/1.db_att/ as per the javadoc of the attachmentprovider class

as an initial hint, to reconstruct the attachments you would need to use mimemultipart , create the mimebodypart objects starting with a query like:

"""select filename, mimetype, size, contentid, contenturi, encoding
from attachment
where messagekey = ${msgkey}"""

addresses

the script provides one utility method to convert a string-ified list where records are separated by the soh character (ascii 1) and email addresses and their corresponding display name are separated by the stx (ascii 2) character.

the processing consists of iterating through the rows of the message table result set, for each row the message body is obtained from a separate sqlite database and then a javamail mimemessage is constructed and output to a file.

troubleshooting

the script cannot cause data loss on your device as it is operating on a backup. if you want to experiment first you may like to set a limit (e.g. limit 10 ) on the first select query to reduce the number of messages that it retrieves.
also, a simple way of viewing the output within e.g. the groovy console is to use msg.writeto(system.out) instead of creating the .eml file.

get the script

the usual disclaimers apply that the script is without warranty, support etc. – get it from github: https://gist.github.com/rbramley/65261127dfb857b03bb6