Provenance Feature

[mobasi-team]

Artifact provenance currently stops at the file in LEAPP's. A report row may point to sms.db, but not to the exact record it came from. To verify a row, an examiner must reopen the database, recreate the module’s query, and find the matching record by hand.

Many modules already know the exact source while parsing. SQLite modules know the rowid. Plist and XML modules know the key path. Modules that decode blob columns may know the row, blob field, and offset. There is just nowhere to store this information in the output.

Proposal: add an optional source locator to each output row.

For example:

• {"table": "message", "rowid": 412} - {"keypath": "..."} - A short path such as row -> blob column -> field
  This could be stored with the source file as (source_file, locator).

The locator should be optional for each row. Some results, such as joined or aggregated rows, do not have one clear source record. In those cases, leaving it out is better than adding an inaccurate locator.

Report writers could include this in TSV or structured output without changing the HTML reports. Existing modules would not need to change. Modules that opt in would become verifiable down to the source record.

If structured output already has a place for this, please point us to it.

Curious what maintainers think. This might intersect with @JamesHabben 's design proposal.

cc @abrignoni @stark4n6 @Johann-PLW

[JamesHabben]

I dont think it changes much with the forensic object design. this is something that could get added right away. the existing source file column from modules is nothing other than the module adding that as a header to indicate the source file path. there is not expected structure and nothing else is consuming it in a way other than it just being a string. it is in a lot of modules now because we have been adding that where it makes sense. so thats where it could feel like its a standard, but its not.

I like the idea of this though. existing source file common practice can stay as is, and we can add a new common practice for source locator. since we dont have any expectation of consuming that value at this point, there doesn't have to be any specific contract on the format. would probably make sense though to put some thought into making a definition for this field.

some of my thoughts on values to store:

• source container type: indicating the parent container file is: sqlite, plist, zip, Protobuf, xml, or others. probably need something like 'proprietary' or binary for the containers unique to a single app.
• data stored type: I am thinking the format of the interpreted data used to build out this row: gzip, plain text, plist, Protobuf ...
• table name: where appropriate like sqlite
• column name: again where appropriate
• rowid: again ...
• key path: for non-sqlite paths: zip, plist, xml
• notes: a sort of catch all to indicate any other special handling of data

so if this record was parsed as an extracted plist, but that plist content was stored as a field in a sqlite table - the source container would be the sqlite with table, column, rowid and then the data stored type would be plist or even NSArchived plist. then it would have a key path for inside the plist as well if needed.