Added 'sort' stage to CI.

[marcfehling]

Part of #389. Separated from #391.

[tjhei]

Do you still want bibtool installed in the docker image or do we leave this as is for now?

[marcfehling]

Do you still want bibtool installed in the docker image or do we leave this as is for now?

I still think it's handy to have in the docker image and in the CI. We can also wait and see how the new feature is received by the community. So there is no hurry right now :-)

[marcfehling]

/rebuild

[marcfehling]

Couldn't call make sort as it would open a shell which is not permitted.

So I tried to copy the instructions. bibtool doesn't like the jabref additions in the beginning and end of the files, so the sed commands are necessary. However, this led me to the Jenkinsfile character-escaping hell and I don't know how to get out.
https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4

[marcfehling]

I found a solution for the sort stage.

Calling make sort from a shell didn't work. I did not understand why. I heard that it was a bug in docker that should have been solved in version 20.10. @tjhei -- Which version of docker do you run on the server?

make: /bin/sh: Operation not permitted

I tried to copy the lines from make sort into the Jenkins stage, but I went insane figuring out how to escape special characters within sed. So I went with an alternative using a regular diff that ignores the first and last lines that contain metainformation, which bibtool can not handle.

Please have a look at this solution. I will squash the commits to one once approved.

Alternatively, we could also setup a github-actions script for this.

[marcfehling]

This is ready for review by the way. I will remove the last commit that checks the CI once approved.

[marcfehling]

ping

[bangerth]

The problem is that it requires contributors to have installed bibtool. What do we gain from requiring people to do that additional step, compared to just doing it outselves every few months?

[marcfehling]

The problem is that it requires contributors to have installed bibtool. What do we gain from requiring people to do that additional step, compared to just doing it outselves every few months?

For the library we do it in the same way with the indent checks, so you can apply the same arguments, too. Continous code maintenance. Detecting duplicates early. Users can check early if their entry is valid (i.e. missing commas). Keeping files readable, as entries are sorted in alphabetical order. Less conflicts in concurrent pull requests: in the past users added entries exclusively in the beginning or end of files, which required frequent rebasing.

bibtool is well-established and available in repositories for basically any UNIX operating system as documented in the readme. Users who are not comfortable with the tool can still inform us about their publications with an issue or an email as proposed in the readme. Others have done it like this in the past, see for example #358 #283 #280 #265.

The additional stage in the CI will issue a report and informs users on how they need to change their entries. So even when users can't run bibtool, they can adopt the changes from the report.

We can teach other users that management tools for bib(la)tex bibliographies like bibtool, bibcure, and biber --tool exist. @gfcas for example got to know bibtool after we introduced it with make sort in #391. He continued to apply it to his own resources, and even proposed a similar patch like this one in #467.

The price however is, and this applies also to the deal.II library itself, that such an indentation requirement adds additional steps for the user, of which installing the tool is the most tedious, but luckily it remains simple.

This check does not need to have a 'required' flag, so we can potentially still merge it when the user fails to sort their entries. This way this CI stage could simply inform us that something is wrong in the publication list. We can also introduce it as an entirely separate stage in the CI, so that a misaligned bib file does not affect subsequent stages. @tjhei -- What do you think of this approach? I can update this PR accordingly.

[bangerth]

Let's see what others think -- opinions?

[marcfehling]

Any opinions on the matter?