Pluggable output formats

[pombredanne]

Following discussions with @sschuberth in #375 we should have a simplified and better to handle output formats. The outline would be this:

1. the only standard format for output would be JSON
2. through a simple plugin mechanism and/or conventions, this could be transformed into other formats such as:

• SPDX Support SPDX tag-value as an output format #338
• HTML "File Information" table present in HTML output even if -i is not given #375 Deprecating plain HTML output  #324
• HTML-APP html and html-app: links should default to opening in a new tab #320 On the html-app license summary, I want to click on the listed licenses and ... #271#154 Check the tree view for long path (deep path) and long file/directory name on Windows #16
• CSV Upgrade json2csv.py script to handle the new ABCD format. #339
• a database (from @yayahalom5776 in Option to write scan result directly into a database scancode.io#1421)
• other formats as needed.

[pombredanne]

So for plugins basics there are a few contenders:

• pluggy (used in in py.test) https://github.com/pytest-dev/pluggy
• stevedore (used in openstack) https://github.com/openstack/stevedore
• pluginbase (from Flask author) https://github.com/mitsuhiko/pluginbase
• roll our own simple stuff with setuptools entry points

All have some pros and cons. In all cases, combining these with setuptools entry points (even if built in) is the way to go IMHO.

[sschuberth]

Creating other formats as a post-processing step from JSON also has the advantage of easily getting scan results in different formats from the same scan. Currently, if you would need both JSON and SPDX, you would need to run the scan twice.

[pombredanne]

Note: A plugin architecture is what is needed here. Some refinements on the formats is welcomed but is not the purpose. The purpose is to refactor the existing code for plugins.

[pombredanne]

and each plugin for format should ideally be also runnable as its own CLI, with input = scan json (e.g. ABC Data format), output = other format

See for ABCD:
https://github.com/nexB/aboutcode/blob/a84e207bbe579dd9b419cc52bbfd00239f5beeb9/aboutcode-data/README.rst

[pombredanne]

So format plugins have landed in develop thanks to @yashdsaraf with #639
There is no new features there for now except that there is a new CSV output plugin.
Beside this we need documentation on how to create plugins and likely implement the JSON-> other format conversion as discussed here.

[pombredanne]

This has been completed and merged in develop.

@sschuberth you wrote:

Creating other formats as a post-processing step from JSON also has the advantage of easily getting scan results in different formats from the same scan. Currently, if you would need both JSON and SPDX, you would need to run the scan twice.

Just ping as if we support #789 in the future, then we would need to have one option for each format such that we can specify which file each output should be written to.

[sschuberth]

we would need to have one option for each format such that we can specify which file each output should be written to.

Or, preferably, extend --format to take a comma separated list, like

--format json,spdx-tv