Report newly added CVEs/vulnerabilities

[nate-droid]

As per a discussion on gitter:

It would be nice to be able to get a diff when polling for new data. This way, when you call the vulnerablecode service, you can easily see which CVEs have been introduced since the last query. This could also look like a hook/trigger that could potentially call another service.

For example, "as a software engineer, it would be nice to be notified when a new CVE is introduced".

Below is the initial question from gitter:
Hey! Been playing around with vulnerable code for a bit, and was wondering if there is a feature to notify when there are new CVE’s? For example, when I make an API call to my vulnerable code service, is there a way to see what info is brand new, rather than fetching all of the data or a specific ID? Essentially, I am hoping to poll my vulnerable code service and be notified if there are any new additions. Thanks!

[jlarfors]

I am working together with @nate-droid on integrating vulnerablecode - it's an exciting project btw :)

It would be enough for us to simply provide a modified date filter or similar to the API. That way, if you run the update every day you could also poll the vulnerablecode API once a day setting the modified date to anything since a few days ago.

Currently now it does not seem possible to know which vulnerabilities where created by the initial import, and which came in later.

[sbs2001]

@nate-droid thanks for the feature request. IMHO it seems essential to have this for integration with other projects. Expect it soon :)

[jlarfors]

That's great, thanks for the fast response @sbs2001 ! And to help us plan a little, what is "soon"? Wondering if we should invest in a workaround for now or just wait. Of course we could try to help you implement this as well, but we are both new to the code base.

[sbs2001]

@jlarfors I have to start cleaning up some tickets here. So 1-3 weeks would be a fair estimate.
@nate-droid btw could you repaste the chat here please, so it doesn't get lost.

[nate-droid]

Just thinking off the top of my head, maybe we could add a column to one of the tables called something like dateAdded, and have a new endpoint called something like vulnerabilities/since/{date} and one would provide the timestamp of the last time you fetched. Anything past that date would get returned.

Taking a quick look at the codebase, perhaps that could be achieved in _get_or_create_vulnerability?

[Hritik14]

@nate-droid Thank you for the feature request.

@jlarfors

Just thinking off the top of my head, maybe we could add a column to one of the tables called something like dateAdded, and have a new endpoint called something like vulnerabilities/since/{date} and one would provide the timestamp of the last time you fetched. Anything past that date would get returned.

The internal structure is going for a big change and the new structure does support having all those dates. After implementing the new model, we'll roll it out in the API as well.
Working on the modeling here: #525
https://github.com/nexB/vulnerablecode/blob/1834ef0fb577ecc8509bdf8f561f48235ca303ec/vulnerabilities/models.py#L322-L327

[DennisClark]

This issue continues to need analysis and action. In the VulnerableCode API.yaml document:

openapi: 3.0.3
info:
  title: VulnerableCode API
  version: 36.0.0

one can find multiple date fields (created, updated, added, etc.) that apply to the original publication of vulnerability-related postings, but dates that apply to the actual posting of the information to the VulnerableCode database are not there (or not obvious anyway).

Also, it does not appear that any of the available date fields can be used as filtering parameters in the documentation at
https://vulnerablecode.readthedocs.io/en/latest/api.html#api-endpoints-reference

So we are a long way from providing history-sensitive API queries.