Test old reports on compatibility

[mhansma96]

Old reports have a different JSON format and within the import-fuction all kinds of things will give errors because some keys don't exist in exports from the old tool

[mhansma96]

I've made some changes that should make it possible for old reports to be imported. But I do lack some old reports or older knowledge of potential scenario's that can fail. We should try to test a few more reports on their compatibility with the new tool after an import. Preferably reports from the old tool. @hidde @nitedog

[hidde]

Great, thanks for those changes, Michel!

We should try to test a few more reports on their compatibility with the new tool after an import. Preferably reports from the old tool.

Agreed. @nitedog do you have any old reports we could use for testing this?

[nitedog]

No, unfortunately not anymore. Maybe EricV can get some anonymized reports from ACC?

[hidde]

Michel and Hidde discussed: Hidde to ask around for old reports.

[ghost]

hey there 👋

For what it is worth:

I have tested backward compatibility with the examples provided in the docs here: https://github.com/w3c/wcag-em-report-tool/tree/master/docs/examples

To test compatibility with previous versions you dont need full reports allthough it is nice to have for more realistic testing. Things to check are:

• Criteria with changed hash urls (id's) wcag 2.0 <> 2.1
• Nested results (sample / page level observations in step 4)
• Simple data like scope, explore etcetera.

There are 2 previous versions viewable online:

• 1.1.0: https://w3c.github.io/wcag-em-report-tool/1.1.0/#/
• current live version: https://www.w3.org/WAI/eval/report-tool/#!/

You can easily create a simple export with each version if you require quick sample data and test compatibility with this version of the tool.

To prevent possible future conflicts I have set up exporting with the jsonld.js API, so at least the json-ld syntax will be compliant. This also helps importability since you should be able to frame any required earl data (assertions, tests, subjects)

All data compatibility checks are done at

wai-wcag-em-report-tool/src/stores/evaluationStore.js

Line 160 in 1ab52d8

async open(openedEvaluation) {

Object @contexts are created in model.js files like in https://github.com/w3c/wai-wcag-em-report-tool/blob/main/src/stores/earl/subjectStore/models.js which affects what data is exported and matched against from imported / opened json-ld files. This means json-ld @context

NOTE: Important though is to be consistent with @context defined IRI's like "WCAG21": "http://www.w3.org/TR/WCAG21/#" and keep using http protocol there. Unknown json-ld data (used for importing observations at step 4) should have their json-ld @context mapped / resolved to using the same protocol, otherwise there will be mismatches (http !== https)

I recommend, if you have not already, to read and play with json-ld to get a little understanding on how the tool's data is structured:
https://json-ld.org/

Hope this will help.

[hidde]

Thanks @rvantonisse for all this context!

We've been able to access some real life old reports and tested with those.