Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix issues in AptSources artifact and support deb822 format (#2851)
# Linux.Debian.AptSources The Linux.Debian.AptSources artifact has a number of limitations, all of which the changes in this pull request aims to resolve. The new artifact also provides parsers that can be imported in other artifacts for inspecting every single field in these configuration files. ## Resolved issues The artifact does not work on current Debian-based systems. One reason is an increased usage of the option "signed-by", used to manually specify the gpg key file location. The original regex only supports the option "arch" or no options. The main reason is the that Release files have been replaced with InRelease files. The format is similar, just with a gpg key embedded at the end of the file. Note that these keys often contain a gpg header "Version: […] that confuses the original VQL. **Fixes and enhancements**: - Support for sources files (multi-line [deb822 format](https://manpages.debian.org/bookworm/apt/sources.list.5.en.html#DEB822-STYLE_FORMAT)) - Support options other than "arch" in list files - Support for any valid usage of whitespace in list files - Support for comments in (at the end of) list files - Support for protocols other than http/https in list files - Support for multiple values in fields/options (arch, lang, targets, components) - Parse InRelease files as well as Release files, and prevent the "Version: GnuPG" header line from interfering with the "Version" field - Group the query results by cache file, preventing duplicate entries without records After improving the list file parser, I also added a parser for the new file format, likely to replace the one-line list file format in the future: deb822. This file format allows each entry to have multiple values (deb + deb-src, multiple URIs and options), and each file may contain multiple entries separated by blank lines. GPG keys may be embedded, and the format is very lenient on whitespace usage and allowing multi-line values. The parser should handle any kind of strange, but valid, formatting, like ``` Key: Value Key2 : Value Key3: # Comment # Comment Value ``` and so on. Due to limitations in the go regex library (no support for look-arounds) and the complexity of multi-value fields and options, the parser VQL is complex and split in a number of functions. Columns are dynamic in order to support every known (and future/unknown) options. This has pros and cons, but it is future-proof. Every known option name is normalised to the names referred to in the man page, which are camel-cased (pascal-cased) and plural. ## Considerations The main motivation behind improving this artifact was to have a working apt sources parser. Since this artifact is part of velociraptor and not the exchange (velociraptor-docs), I've tried to make the parsers handle everything I can throw at it. I am happy to include tests, but before I do that, I would like to get some feedback. I kept the original query, but I also included two others. I feel that a table including every parsed field, like all options, is handy. Likewise, a flattened version is also very useful to work on in a notebook. This results in two rounds of parsing sources files (due to how flatting happens at an early stage), and the third query parses cache files as well, like before. This shouldn't have any noticeable effect on any normal system, but if there are ways I can rewrite the VQL do only parse the files once, for both the combined and flattened results, that would of course by much nicer. And perhaps something belongs in notebook suggestions? I see lots of potential in this artifact, but for starters, I'd like it to work and be useful for importing in other artifacts. Working with inrelease-path and signed-by options in future enhancements may be useful. I find it a bit tricky to get an overview of all the supported features of artifact YAML, so feel free to point out how I may improve the documentation, ways to export functions (without including all "private" helper functions), and how to improve the current three–query solution. --------- Co-authored-by: Mike Cohen <mike@velocidex.com>
- Loading branch information
