Duplicate extracted licenses when parsing from RDF

[xavierfigueroav]

The field Document.extracted_licenses contains duplicate ExtractedLicense objects when they are parsed from RDF files.

It can be noticed by running parse_rdf.py.
Input: SPDXRdfExample.rdf
Output:

doc comment: This is a sample spreadsheet
Creators:
        Person: Gary O'Neall
        Tool: SourceAuditor-V1.2
        Organization: Source Auditor Inc.
Document review information:
        Reviewer: Person: Suzanne Reviewer
        Date: 2011-03-13 00:00:00
        Comment: Another example reviewer.
        Reviewer: Person: Joe Reviewer
        Date: 2010-02-10 00:00:00
        Comment: This is just an example.  Some of the non-standard licenses look like they are actually BSD 3 clause licenses
Creation comment: This is an example of an SPDX spreadsheet format
Package Name: SPDX Translator
Package Version: Version 0.9.2
Package Download Location: http://www.spdx.org/tools
Package Homepage: None
Package Checksum: 2fd4e1c67a2d28fced849ee1bb76e7391b93eb12
Package verification code: 4e3211c67a2d28fced849ee1bb76e7391b93feba
Package excluded from verif: SpdxTranslatorSpdx.txt,SpdxTranslatorSpdx.rdf
Package license concluded: LicenseRef-4 AND LicenseRef-2 AND Apache-1.0 AND LicenseRef-3 AND LicenseRef-1 AND Apache-2.0 AND MPL-1.1
Package license declared: MPL-1.1 AND Apache-2.0 AND LicenseRef-3 AND LicenseRef-2 AND LicenseRef-4 AND LicenseRef-1
Package licenses from files:
        LicenseRef-1
        LicenseRef-3
        Apache-1.0
        MPL-1.1
        LicenseRef-4
        LicenseRef-2
        Apache-2.0
Package Copyright text:  Copyright 2010, 2011 Source Auditor Inc.
Package summary: SPDX Translator utility
Package description: This utility translates and SPDX RDF XML document to a spreadsheet, translates a spreadsheet to an SPDX RDF XML document and translates an SPDX RDFa document to an SPDX RDF XML document.
Package Files:
        File name: Jenna-2.6.3/jena-2.6.3-sources.jar
        File type: ARCHIVE
        File Checksum: 3ab4e1c67a2d28fced849ee1bb76e7391b93f125
        File license concluded: LicenseRef-1
        File license info in file: LicenseRef-1
        File artifact of project name: Jena
        File name: src/org/spdx/parser/DOAPProject.java
        File type: SOURCE
        File Checksum: 2fd4e1c67a2d28fced849ee1bb76e7391b93eb12
        File license concluded: Apache-2.0
        File license info in file: Apache-2.0
        File artifact of project name:
Document Extracted licenses:
        Identifier: LicenseRef-4
        Name: None
        Identifier: LicenseRef-2
        Name: None
        Identifier: LicenseRef-3
        Name: CyberNeko License
        Identifier: LicenseRef-1
        Name: None
        Identifier: LicenseRef-3
        Name: CyberNeko License
        Identifier: LicenseRef-2
        Name: None
        Identifier: LicenseRef-4
        Name: None
        Identifier: LicenseRef-1
        Name: None
Annotations:
        Annotator: Person: Jim Reviewer
        Annotation Date: 2012-06-13 00:00:00
        Annotation Comment: This is just an example. Some of the non-standard licenses look like they are actually BSD 3 clause licenses
        Annotation Type: REVIEW
        Annotation SPDX Identifier: https://spdx.org/spdxdocs/spdx-example-444504E0-4F89-41D3-9A0C-0305E82C3301#SPDXRef-45

[goneall]

@xavierfigueroav Please attach the full output and the input files to the issue.

[xavierfigueroav]

@goneall I've added a link to the input file now.
I had already included the full output produced by parse_rdf.py. What do you mean exactly?

[xavierfigueroav]

In the input file, there are 4 extracted licenses inside some <licenseConcluded> tags. The same 4 extracted licenses are referenced (<member rdf:nodeID="ID"/>) inside a <licenseDeclared> tag.
Is that something wrong in the rdf file, something that should not happen? If yes, there was never any error in parsers. If not...
Both extracted licenses in <licenseConcluded> and extracted licenses in <licenseDeclared> are being added to Document.extracted_licenses by the following code (specifically, line 265):

tools-python/spdx/parsers/rdf.py

Lines 264 to 267 in db243e3

	if (lics_member, RDF.type, self.spdx_namespace['ExtractedLicensingInfo']) in self.graph:
	lics = self.handle_extracted_license(lics_member)
	if lics is not None:
	licenses.append(lics)

That code can be split into a separate method with some changes for it to add the extracted licenses children of <SpdxDocument>. Maybe:

for lic, _, _ in self.graph.triples((None, None, self.spdx_namespace['ExtractedLicensingInfo'])):
    self.handle_extracted_license(lic)

I am no so familiarized with the SPDX RDF representation (e.g., I don't know whether all extracted licenses are always direct children of <SpdxDocument> or not), so this may be working on this and some files, but it may not be a general solution. If you think it is OK, I can fix it that way and submit a PR.

[goneall]

I took a look at the RDF file and it looks valid.

The 4 extracted licenses are only defined once in the RDF and the same license is referenced in other parts of the document (the ID is used to reference the same license definition).

@xavierfigueroav It looks like you found the source of the problem above - it should not add the extracted license info on each reference, it should only add it once.

Your proposed solution is very similar to the approach we take in the Java tools - adding the extracted license information as a separate method.

Just a bit more context if interested (probably not needed to fix this specific bug) - for the RDF, there are 3 ways of referencing an object - a literal value (like a string or number), a URI to point to a predefined object and Anonymous Nodes which are references generated and are local to the RDF graph. For extractedLicenseingInfo definitions Anonymous Nodes are generated and referenced throughout the RDF graph. A good overview can be found at https://www.w3.org/TR/rdf-concepts/

[xavierfigueroav]

@goneall Thank you for the link, it was and will be helpful.