Create PURL services CLI tool and library

[pombredanne]

To best support using various PURL-based services, I would like to have a command client tool and library as a client API that can expose these services for integration elsewhere.

[johnmhoran]

@pombredanne @AyanSinhaMahapatra

I've looked at the SCTK fetch_thirdparty.py example, but I have to admit that I don't understand what a complete command for that utility would look like or how it might apply to the current issue. Examples of how to run the fetch_thirdparty example would be helpful for me to explore how that works. (I've looked but found no documentation/examples for that utility.)

In addition, the description above of the current issue seems rather vague. What does it mean to create a client API tool to access PURL services? Examples of PURL services we want to handle, and some descriptions of user input and output, would be particularly helpful.

The only exposure I've had so far with the PurlDB is the experimentation I've done since last Friday evening with the new validate endpoint.

• Is that an example of a service you want this CLI tool/library to handle?
• Would we want, for example, a command-line function that enables a user to input -- through the command line or perhaps a .txt or .xlxs file -- the 1+ PURLs that he or she wants to vet with the new validate endpoint? Maybe with options of terminal output and a .xlsx workbook?

[johnmhoran]

@pombredanne Now that we've (initially) addressed the validate endpoint with our new CLI, what additional "services" do you want me to focus on, and how can I identify them and begin to understand how users use those services?

[johnmhoran]

@pombredanne As noted last week, I'm blocked for now from additional CLI work until we can add the missing details to your initial description of this issue, i.e., ID the additional services, commands and use cases we want to include.

[pombredanne]

So the next steps after validate (and after adding tests to validate) would be to use the latest and new fetchcode as a library to add two new sub commands:

• versions: given a PURL, return the list of all known versions for this purl
• meta: given a PURL, return a mapping of metadata fetched from the API for this PURL

After this I would like to see these:

• urls: given a PURL, return a list of [{URL type: URL}, ...] as in [{"homepage_url": "https:example.com"}, {"vcs_url": "...."}] and various download URLs. Use the packageurl library for this (purl2url) and this will need updating as needed, and use as well scancode-toolkit packagedcode or code in dejacode. Optionally validate each URL existence with a head request (ask @chinyeungli on how to do this)
• purlcli: purl2scan #277 scan: given a PURL, fetch the URLs and call a scancode.io API and run a scan with a scan_package pipeline, then return the scan results. Either wait for the scan to complete or poll until completion. Later implement the same with PurlDB which has code in the "priority queue" to handle this.
• purl2git: From PURL get Git versions and tags  #258 git: takes a PURL and returns a new PURL for the corresponding git repos and the tag matching the PURL version. And this for the implemented Maven, npm and Debian package types.

[johnmhoran]

@pombredanne Re the first bullet above -- a versions subcommand based on fetchcode -- are you looking for this sort of output, or perhaps just a list of versions as strings? (This is an excerpt from the pkg:pypi/scancode-toolkit output.)

        purl_versions = [
            [
                PackageVersion(
                    value="2.0.0",
                    release_date=datetime.datetime(
                        2017, 6, 23, 8, 35, 20, 322426, tzinfo=tzutc()
                    ),
                ),
                PackageVersion(
                    value="2.0.0rc3",
                    release_date=datetime.datetime(
                        2017, 6, 16, 16, 24, 2, 443222, tzinfo=tzutc()
                    ),
                ),
. . .

[johnmhoran]

Compare just the versions as strings, e.g.,

results_values = ['2.0.0', '2.0.0rc3', '2.0.1', '2.1.0', . . . '32.0.5rc3', '32.0.6', '32.0.7', '32.0.8']

[johnmhoran]

@pombredanne Do we want to get version data for both one PURL and for multiple PURLs, depending on the user's need? (Just as we do with validating either a single PURL or a list of PURLs.)

Also: What should the output look like: a list of string versions, or JSON (and if so, what would it look like)?

[pombredanne]

Always start with a single PURL. Expanding to a list is easy.

The output could be either:

1. something like {"purl": "... input purl", "versions": ["1.1", "2.3" , ....]}
2. or may be better: {"purl": "... input purl", "versions": [{"purl": "pkg:...@1.1.2", "version": "1.1.2"}, {"purl": "pkg:...@1.1.3", "version": "1.1.3"}]}

This will account for multiple PURLs in both cases.

Eventually the output will need account for the input in some header instead, much like in a ScanCode scan, but this is for the future, but nothing urgent for now.

[pombredanne]

Manage objects internally, and deal with simple/plain serialized Python data at the end only.
Adding the release date of each version works too BTW, just make sure you use an ISO timestamp like it is done in our other APIs.