Option to write scan result directly into a database

[yahalom5776]

I think it would be great to have the possibility to hook the scanner directly to a database (e.g. PosgreSQL or similar) and write its output into the dabase. Also, the possibilty to specify tags like ID, component name and version with each scan would be very helpful.

[pombredanne]

@yahalom5776 This is an excellent suggestion... and in fact this is coming up in two forms:

• I have a prototype of a web-based scanner that I should push to a repo soon
• @jdaguil has been working on an Electron-based desktop app to more efficiently review and draw conclusions from scan results and in particular component names and versions

Also, the possibilty to specify tags like ID, component name and version with each scan would be very helpful.

What do you mean by ID?

[yahalom5776]

@pombredanne Thank you, great news. Appreciate it. With ID I mean some kind of internal unique identifier that could be used to map scan results from different tools to one OSS component for example.

[pombredanne]

@yahalom5776 with the upcoming package detection you would get a package id... but not a what I would call a component id. For instance Apache CXF is a component... it has eventually many different packages for each given releases (for instance all the Jars available on Maven) or Rails (Ruby) is a component, but it is effectively many Gems. So while we can report packages, components are a higher order concept that requires curated data in most cases...
Does this align with your idea of a component?
And when you mean a component id, did you mean a package id? or some manually assigned package?
Also which other tools would you need to reconcile ScanCode data with?

[yahalom5776]

@pombredanne You are right, when I wrote component ID I meant an unique identifier for one package that I scan (which would normally be an archive). Other tools involved in the process might be separate scanners (not only for OSS but for example also for security vulnerability detection) or management systems like a repository (e.g. Artifacotory). One possible solution I came up with would be to use the hash of the package (archive) as an identifier. But I am not entirely sure yet, have to think about this a little more.

[pombredanne]

@yahalom5776 a hash is not a bad approach per se, but it depends which one: for instance a SHA1 or similar would be very brittle to even the most minute change. What is your code mostly about tech-wise? Is this mostly Java (you mentioned Artifactory)?

[pombredanne]

@yahalom5776 you mentioned security vulnerability detection .... the only safe way I know of ID-wise for this is to use the CPE as an id for this kind of "id mapping" at least in the context of the NVD.

[yahalom5776]

@pombredanne It would be a little bit of everything tech-wise (C/CPP/Java, C#, etc.). Can you share a little bit more about the web-based scanner and its architecture perhaps?

[pombredanne]

@yahalom5776 FYI, I am starting playing with storing the scan in per-scan temp sqlite dbs .I will push in a branch soon and will ping you then.

[armudgal]

@pombredanne Are you still working on it? I was thinking of figuring how to store the scan in the database, and did you figure something out for the ID thing? Will you be implementing it or will leave it for later ?

[pombredanne]

@Kira0204 Yes I am. The branch https://github.com/nexB/scancode-toolkit/tree/orm-models is were I am working on actually using DB models internally as part of aboutcode-org/scancode-toolkit#275. Though a lot of it hass changed and I did not repush it yet. The DB in itself is not the issue. The data models and structures are. Once that is done, the DB is a natural by product.

ID'ing also becomes a natural thing with a clean modelling and not an issue.

[pombredanne]

@Kira0204 This was marked by mistake as an easy ticket. It is not. It is quite complex and touches the core of it all.

[pombredanne]

We now have ScanCode.io that does all this database storage. I am closing here and moving the issues there too.