Spellchecker for English LP content

[spier]

I added a basic setup for a GitHub Action that runs a spellchecker for the LP content.
This is modelled after a very similar approach that we are just introducing for our Patterns (see InnerSourceCommons/InnerSourcePatterns#519).

A key difference here is that the tool used here, pyspelling, does seem to work a bit better with markdown, than it works with asciidocs. However I tweaked the configuration based on an example that I found, and it looks like it works well enough.

To judge the effect of the spellchecker, please compare the results of the 1st run, with the 2nd run. Before the 2nd run I had already fixed a couple of obvious spelling errors.

Left to do:

• decide if you like this approach
• decide if you want to spellcheck for en_US (so American English). if yes, remove the last block in wordlist.txt and fix the respective spellings (mostly just the word "organisation" 😄)
• improve the config in .pyspelling.yml and the wordlist.txt, so that the spellchecker passes on the current content (adapting the failing parts of the content is of course also another way to fix this)
• decide where to place the config files - they could be placed in some folder, if you don't want them in your root

I am happy to answer your questions. Just note, that I am only using pyspelling/aspell for a couple of days myself now :)

[rrrutledge]

This is great, @spier ❗️ Thanks for this work. Very excited to take a look.

[rrrutledge]

This is great! Where does the output go after the job is run?

[lenucksi]

This is great! Where does the output go after the job is run?

If this were to raise a PR with the fixes it made it'd be great.

[spier]

@lenucksi the spell checker does not make any fixes itself.
Not sure if it can be configured to do that. I doubt it actually, as it would be hard to know what the intended word is.

What it does is that it shows the spelling issues in the output of the GitHub Action.
(@rrrutledge not sure if this answers your question about where the output goes?)

It is then up to the author to review those issues and decide how to fix them. The options are:

1. fix the words that the spell checker identified as problematic
2. add the given word to the list of exceptions

FYI for our patterns repo I am experimenting with another spell checker right now (Vale).
That one would show the spelling issues inline in the files. I can also do other checks for prose to get to a more consistent editorial style (inclusive language etc).

I am not sure yet which spell checker we will go with for our patterns repo.

So if you like, we can move this PR here back to Draft for the moment. Up to you.

[lenucksi]

@lenucksi the spell checker does not make any fixes itself. Not sure if it can be configured to do that. I doubt it actually, as it would be hard to know what the intended word is.

Most mistakes are very likely small errors like swapped out letters. All of that yields a very small edit distance.
I'd expect the spellchecker to have a configurable safe threshold for adding their most likely correction.
And that's exactly why a PR is the way to go here. It proposes changes without any modification that you can then fix or entirely discard ideally saving you some time.

What it does is that it shows the spelling issues in the output of the GitHub Action. (@rrrutledge not sure if this answers your question about where the output goes?)

Having to manually extract edits and a diff here really sounds like more new work created than time saved, so I'd not use such a feature.

As for the style consistency there's still #222 which would address this. There's tooling to address this in a professional and effective way, it's not free though. Knowing those tools they are usually more interactive and less CI friendly, so that's probably not really first order of business - as was the conclusion of #222 as well.

[rrrutledge]

This is very 🆒 , @spier . I will bring this up in our next working group meeting.

I am not sure yet which spell checker we will go with for our patterns repo.

I think that there's value in being on the same one, so let's see what you come up with.

show the spelling issues inline in the files

As comments or something? That would be great.

spelling issues in the output of the GitHub Action

I see it now. Great.

[rrrutledge]

@spier is there an issue or a pull request that we can follow in the InnerSourcePatterns repo for us to stay in touch with what spellchecker you are choosing? We are happy to follow your lead, here. Thanks for including us :).

[spier]

@rrrutledge good idea. here is the issue: InnerSourceCommons/InnerSourcePatterns#519

[rrrutledge]

Thank you, @spier ❗️ I added myself as a watcher.

[rrrutledge]

We should update our CONTRIBUTING.md document to reference how to find and fix spelling errors in pull request.

[marshmallowrobot]

Hey @spier! It seems like progress on this PR has slowed. But, we have a new group of articles that will be published soon and I think having a spell checker in place would be a great help with them. What, if anything, can I do to help get these changes pushed through?

I can see from the conversation in InnerSourceCommons/InnerSourcePatterns#519 that you'd prefer to change the spell checker to Vale. Could you maybe list out the relevant config files (versus ones you may have been experimenting with) from InnerSourcePatterns repo that we should use over here? I could help get the code changes copied over.

[spier]

@marshmallowrobot thanks for reaching out. You are right, there is no further progress on this from my end.

And yes, my last status was that Vale looks like the better option for us, given that the integrations into GitHub seemed nicer. See also the summary of the overall status of the experiments I did with vale.

Thank you for your offer to help!

Unfortunately I forgot where I was at exactly, so I cannot even ask for specific help right now. I have to look at this in more detail again to determine the most minimal set of configuration files that you would need to make vale useful for you. Will let you know what I find, however it may take a bit.

[rrrutledge]

Thanks for this work and this update, @spier ! We are very thankful and excited for spell checking. Let us know if you have a rough idea of when you might work on it. Any schedule is OK, but if you think that it will be a while before working on it some more then we may merge what's here so that at least we have some spell checking in the meantime until your fancy, new Vale work is finished.

[spier]

That is good clarification @rrrutledge.

From previous comments I thought that the current solution would not be that useful for the LP content, which was one of the reasons for trying out vale.

So I will keep the current solution as a fallback option.

As for an estimate, I hope to have time around the Easter days, so within the next 10 days or so should be realistic.

[rrrutledge]

next 10 days or so
Oh! That’s not so long - we’ll probably wait. We’d rather be in the same thing as InnerSourcePatterns.

[rrrutledge]

@spier we saw that you created https://github.com/InnerSourceCommons/isc-styles for some more Vale work - so it looks like this may be moving along? That's great!

[spier]

@rrrutledge hehe you got your eyes everywhere :)

Yeah, I am trying to package up the styles/checks in a central place, so that it is easier to create some level of consistency across various ISC material.

I am not 100% there yet but I have a basic idea of how it might work. So yeah, this is moving along, slowly but surely :)

[spier]

Closing this in favor of #560