OAI-Pmh fails with SAXParseException

[dr0i]

Reported by @hagbeck :

"https://eldorado.tu-dortmund.de/oai/request" |
open-oaipmh(metadataPrefix="oai_dc", dateFrom="2020-09-19", dateUntil="2020-09-19") |
decode-xml |
handle-generic-xml |
encode-formeta(style="multiline") |
write("stdout");

results in:
SAXParseException; lineNumber: 194; columnNumber: 90; Zeichenreferenz "&#55349" ist ein ungültiges XML-Zeichen

[fsteeg]

Here is the example provided by @hagbeck to show the XML should be valid:

https://eldorado.tu-dortmund.de/oai/request?verb=ListRecords&from=2020-09-19&until=2020-09-19&metadataPrefix=mods

In the browser this shows the result of their XSLT transformation, see 'view source' for the XML. However, I think this only works because their XML processor is lenient, because the XML actually declares itself as ?xml version="1.0", but arbitrary Unicode characters are only supported in XML 1.1, see https://www.w3.org/TR/xml11/#sec-xml11:

Finally, there is considerable demand to define a standard representation of arbitrary Unicode characters in XML documents. Therefore, XML 1.1 allows the use of character references to the control characters #x1 through #x1F, most of which are forbidden in XML 1.0.

(I tried to update Xalan in metafacture-biblio to 2.7.2 as suggested by @hagbeck, but that makes no difference – which I think makes sense since there is no XSLT involved in the sample Flux above).

So I see 3 possible approaches:

• Change the ?xml version="1.0" declaration in the XML to ?xml version="1.1" (I suppose that's not so easy, @hagbeck? It also might have unexpected side effects on other users etc. Also, we should first verify that actually solves the issue)
• Tell the Java SAX parser to be lenient or override the 1.0 declaration in the document (a quick research did not show any way to do that, but there are a lot of ways to configure these parsers, so there should be some way)
• Change the incoming XML declaration based on some config of the OaiPmhOpener (like xmlVersion=1.1), resulting in XML with the ?xml version="1.1" declaration being passed to the Java XML parser

[dr0i]

I try to update the OCLC library to use 1.1 instead of 1.0.

[dr0i]

Made the OCLC library to output xml version 1.1. That version was recognized by the AbstractSAXParser where the Exception is thrown. But it didn't change a thing.
Stumbled about https://stackoverflow.com/questions/15634536/java-sax-parser-mangles-attributes-for-xml-1-1 where it is recommended to not use JDK XML parser but to switch to Apache Xerces XML parser.
This should be tried.

[fsteeg]

I don't understand this:

Made the OCLC library to output xml version 1.1. That version was recognized by the AbstractSAXParser where the Exception is thrown. But it didn't change a thing.

It throws the same exception although it's now getting XML 1.1?

[dr0i]

Exactly.

[dr0i]

Ah, nasty nasty, dependency hell ... Getting rid of xalan-serializer, which is a dependency of xalan, but not needed by the OAIPmh, fixes it, see https://stackoverflow.com/questions/11952289/serializing-supplementary-unicode-characters-into-xml-documents-with-java . Will provide a proper config tomorrow.

[dr0i]

It's like this:
if in build.gradle the xalan:xalan:2.7.2 is implemented, I see in the used libraries in the IDE: xalan, xml-apis and serializer. Run fails.
If only the xalan library (shipped with oaiharvester) is used, the xalan appears in IDE, but no xml-apis nor serializer. Run succeeds. I.e., all data can be retrieved, but only when encoding is set to ISO-8859-1, which breaks UTF8-characters (like mathematic symbols). I don't know how to cope with that properly.
As @fsteeg said your oaipmh-server serves xml version=1.0. This seems to be used by the parsers of the oclc-oaipmh (and OaiPmhOpener wraps its own xml-header around the output . One could set xml version=1.1 but this is not used by the parsers, just as xml-header for the output.) I will could to manipulate the xml-source to check this, but maybe it would be even possible for you @hagbeck to set the xml-header to 1.1 ?

[hagbeck]

We will check this.

But yesterday I discovered in an other context, that the current version of xmllint in Ubuntu 20.04 doesn't support the xml version=1.1. It seems that this solution isn't stable enough for all use cases, isnt' it?

[dr0i]

ACK. Also, I just let the OAIPmh ran as standalone. Surprise - it I works perfectly with your OAI-Server ! So it must be some library dependency and this should somehow be solvable.

[dr0i]

@hagbeck try branch 334-fixEncoding, this should work. It basically uses some older libraries and excludes some others explicitly.