Workshop IV

[alee]

This is the start of a discussion on the Reusability Standards for the OMF Workshop IV

https://openmodelingfoundation.github.io/standards/reusability/

Breakout room 4

FAIR4RS RDA goals

The software is both usable (it can be executed) and reusable (it can be understood, modified, built upon, or incorporated into other software).

@SusanCuddy : this goal does not really cover reproducibility - should we keep the reproducibility parts of the Note and maintain it in the goals

@gregtucker : what do we all think about discussing reproducibility in reusability? reusability is easier to meet than reproducibility

@SusanCuddy : for reusability would prefer it to be a black box and just consider its inputs -> outputs

@mdbarker : reproducibility is a higher level claim, something can be FAIR but not reproducible. endless discussions on defining research software

@alee : perhaps we focus on reusability standards and include a sidebar on how reusability facilitates reproducibility

@SusanCuddy : what is our goal for this workshop breakout? Should we clarify the distinction between reusability and reproducibility? Is there a clearly defined value proposition for reuse

@mdbarker : reuse is very important (see comment below) - increase public trust, start with the people that get it and keep going, small work builds up to big work. stop reinventing the wheel, improve economies of scale

@gregtucker : increase efficiency - sometimes grad students pick up idiosyncratic code and struggle to accomplish anything with it over months, and sometimes they pick up a well designed modular set of code and do something remarkable with them. Also there is something to be said about the care and design that reproducibility requires that improves reliability in general

@hsu000001 : one more thought on why we are striving for reusability - there are some that are very interested in generating things like "quick start notebooks" or "click to run on compute resources" from existing models, or helping that to be more common in the future.

Breakout room 1

Andrew Moore: should the reusability standards depend on the accessibility standards? consider the case of research software thrown over the wall (with well documented API)

versioned input data needs clarification (sample data) #288

Mary Uhlmansiek: include the OS of closed source products in the errata

Breakout room 2

does reusability cover replicability ?

luciana: keeping reproducibility separate from reusability - reusability is a good minimal standard

sean barton: models are built on theoretical frameworks, context is important - that should be independent of the source code. Plenty of times when source code was there but reproducibility couldn't be done due to missing context

consider reproducibility as something to be elevated into its own standard eventually, or that the other standards lead to reproducibility

George Percival: we should be welcoming of models as a service, well documented and defined APIs. Craft some language to establish standards for web accessible models as a service, favor open file formats, network API protocols

Language on acceptable licenses?

Black box reuse - everything happens on the interfaces, so the interfaces should be well known and tested.

ideal standards -> gold standards in the medical world

Breakout room 3

Tony Jakeman: create a minimal standards checklist that includes all the standards in a terse / easy to consume format #167
standards should inline linked bullet points when necessary instead of saying "meets minimal standards from Accessibility etc"

how do we address data quality? both in inputs and outputs

https://wesharedata.org/

anyone can claim that their model meets the standards, how to verify and authenticate claims for meeting the standards?

one possibility: authenticated badging from certified / verified organizations (who have passed a train the trainers regimen)

Rebecca + Cheng-zhi: There is a lot of overlap between accessibility and documentation and reusability. We should define the terms clearly and only list the standards relevant to the definition of that term.

[mdbarker]

Why reusability (and reproducibility is important)

• enhances trust in science by decision-makers and the public, potentially increasing funding for, and application of, research outcome
• reassures universities who are concerned about reputational damage from recent article retractions
• helps grow community around a model, adding possible developers, users, funders etc - to demonstrate impact of model and increase sustainability