"File Information" table present in HTML output even if -i is not given

[sschuberth]

Scanning for file information (-i) is disabled by default, yet the (empty) "File Information" table is included in the HTML output. I was looking into fixing this, but I could not get my head around where the actual information is added to the info dictionary. In here only the file name seems to be added, but where is the md5 etc. added before feeding it to the HTML template? I know the info is calculated here, but I do not see how this gets fed into the HTML template.

[pombredanne]

@sschuberth in the latest code for #267 now mostly in develop the file information is always collected and reported in the final JSON only if effectively requested. This is because the info is used for the cache keys generation. Now the nature of how the data is store for info has also changed and this is no longer a separate data block for a file but instead plain attribute at the file level which probably added to the confusion.

Now, the html output (the the html-app) is entirely driven by a Jinja template found here https://github.com/nexB/scancode-toolkit/blob/ca640b92d9cb82c3523aa229dd26c76907b1a17b/src/scancode/templates/html/template.html and this has likely not been updated correctly when info have been rolled up as file attribute.

I hope this makes sense for a start.... more to come tomorrow

[sschuberth]

Thanks @pombredanne, I'm aware that you use Jinja to fill in the template. But I don't get where the row.type etc. are filled in the Python code. I do get that row.name comes from here, but what about all the other file info properties?

[pombredanne]

@sschuberth you stumbled on the kludgy part... It felt not right when I wrote this and I knew this would boomerang: you have eagle eyes!

The info are no longer under an explicit infos attribute: instead they are each a direct attribute of a file.
And they are not collected explicitly but instead implicitly by exclusion.

So in this loop https://github.com/nexB/scancode-toolkit/blob/ca640b92d9cb82c3523aa229dd26c76907b1a17b/src/scancode/format.py#L207

        # this is klunky: we need to drop templates entirely
        converted_infos[path] = OrderedDict()
        for name, value in scanned_file.items():
            if name in (LICENSES, PACKAGES, COPYRIGHTS, EMAILS, URLS):
                continue
            converted_infos[path][name] = value

... if a file-level attribute is one of the "other" scans e.g. one of (LICENSES, PACKAGES, COPYRIGHTS, EMAILS, URLS) it is skipped with the continue.
Otherwise it is considered by exclusion as a being a direct file-level info attribute and added implicitly with
converted_infos[path][name] = value to the "info". This is confusing: it took me a while to understand what I wrote there :|
So having an explicit list would likely be much better than an exclusion, implicit list.

With all that said, I think this is an area that could be simplified in the future:

• formats could be dropped from the core entirely keeping only JSON
• with a simple plugin architecture, multiple format could each do their own cooking and eventually be called alright from the main CLI (either as a an option like today of a git-style sub-command) or with they own dedicated CLI.

This would make it cleaner to handle existing and new formats (e.g CSV, SPDX, etc)
What do you think?

[sschuberth]

I agree to what you say about the output file formats. I was also wondering why not to use only JSON as the (primary) output, and implement all other formats as post-processors of it. Also, there probably should not be a separate JSON flavor for the html-app anymore, it should simply use the "primary" JSON format as-is.

[pombredanne]

agreed! On the html-app side, the only thing that it needs extra is the variable name to make it a JSONP data file, but that can be added after. This is unfortunate but @jdaguil and I could not find a way out. Note that the new electron-based app at https://github.com/nexB/aboutcode-manager reads the default JSON directly (and only that).

[pombredanne]

@sschuberth unless you feel strongly about this and need this I would like to close this as wontfix. I am not too keen to have more work done on the HTML side.