Charlie Freestone Blog Post - Some Best Practices for Accessible Automation Testing #317
Conversation
There was a problem hiding this comment.
Overall, plenty of good advice.
I was surprised that an article about testing with "Accessible" in the title only lightly touched on accessibility concerns - using roles to locate elements. I feel this is somewhat misleading, as the rest of the article concerns your thoughts on naming and tagging tests to make them more informative, plus the part which I think is about not worrying too much about specificity, not testing in too much detail the things which are not all that important, none of which is about accessibility.
I would advise changing the title to reflect this.
| categories: | ||
| - Testing | ||
| --- | ||
| # Some Best Practices for Accessible Automation Testing |
There was a problem hiding this comment.
Please remove this one, as your title is provided for you:
| While on my most recent project I had the unique experience of working closely with many testers and test minded individuals. This allowed me to learn some much-needed lessons about how to best implement automation testing with accessibility in mind, a sometimes-overlooked area of test automation. | ||
|
|
||
| ## Introduction | ||
| What I hope to share with you is how simple it can be to both think about and implement the approach to make sure your tests are accessible. This can be because you have strict reasons on the project to ensure compliance with certain metrics, because this can lead to improved readability of your code and its longevity, or because you want to read some wonderful thoughts from certain Testers perspective. |
There was a problem hiding this comment.
certain Testers perspective
Not sure what you are trying to say... Do you mean "from a certain tester's perspective", i.e. you? Or did you mean plural "testers"?
I think this introductory paragraph highlights my main question about your article: what does it mean for a test to be accessible, and what do metrics or code readability have to do with that? Are you concerned about making your test code accessible to testers who need to use a screen reader? If so, I think you need to make that clear, and give more examples of how you can achieve that.
There was a problem hiding this comment.
Certain testers perspective was meaning myself.
What I wanted to share was skills I learnt which anyone can write in their tests which is in line which accessibility testing. Its not aimed at testers who need a screen reader. Its lessons I have learnt from doing Accessibility testing, which can make your automation tests more accessible.
There was a problem hiding this comment.
Certain testers perspective was meaning myself.
Then you want "from a certain tester's perspective" (need apostrophe)
Accessibility testing is about testing that a UI is accessible to all users, whether they are keyboard-only users, have vision or motor impairments, dyslexia, epilepsy, etc. The only part of this article which is about accessibility is locating elements by role. You are not really making your automation tests more accessible, you are making them more readable.
There was a problem hiding this comment.
By the way, how did you do Accessibility testing, what tool(s) did you use? That would also make an interesting blog post.
_posts/2025-05-14-some-best-practices-for-accessible-automation.md
Outdated
Show resolved
Hide resolved
|
|
||
| This neat little function is the bread-and-butter basics of automating in playwright, it's what we want to see when we place our test into a certain configuration. This doesn’t mean it has to do handstands or backflips, but just like any gymnastics routine we do wait on it with bated breath, hoping for results we will cheer for. This anticipation makes it easy for us to write something which makes us wait for that big finale in our tests, even if the tests took the literal definition of breaking a leg. | ||
|
|
||
| And therein lies the issue; during a test we usually assert for confirmation to ensure we are where we expect. That means we can `expect(response.status()).toBe(200)` which is a wonderful successful request that has found the requested resource of a page. Instead we get a `201 response`, which is just as equally a successful request but it led to the creation of a resource. Test results can’t always be perfect and that means that this stopping point during the test, before we check the larger critical parameters later on, will skip the test. Instead, we have to accept that not every response will be exciting and just `expect(response.status()).toBe(OK)` which allows us to still capture data from a test even if the status code is different. |
There was a problem hiding this comment.
What is OK here? I can find no reference to it in the playwright docs. I think you mean https://playwright.dev/docs/api/class-apiresponseassertions#api-response-assertions-to-be-ok
While I think I understand what you are trying to get at here (be generic, we just need a successful response) it is potentially confusing because if we expect something to be created then we can legitimately expect 201, whereas if we are simply Getting something, then we can expect 200. Are you arguing that 200 and 201 are less readable than OK? You are losing test precision when you use toBeOK, so are you arguing that readability trumps precision?
There was a problem hiding this comment.
Yes that was an error it should be "expect(response.status).toBeOK()" which I can edit.
I am saying that that at a point in the test we can check its "Ok" before we look at another parameter in that same test that comes later on in that test.
For that specific test we are less concerned on if it is 200 or 201 but instead on the result of the overall test. So, make the test more readable and use "expect(response.status).toBeOK()".
There was a problem hiding this comment.
That is a matter of opinion! Developers get very particular about RESTful APIs, and if you are creating something then the API should return a 201. You will probably have API tests covering that though, so the more readable toBeOK() is fine.
|
|
||
| So, when creating Test Names its best practice to instead use the back tick ` as this will then still allow you to use a variable in the test name as seen in this example: | ||
|
|
||
| ```test(`should return a 400 when X-CUBE-ASSERT is '${user}'`)``` |
There was a problem hiding this comment.
While the use of backticks when there is no variable to interpolate is somewhat contentious, of far more importance here is that the name of the test contains implementation detail: X-CUBE-ASSERT which is presumably a header. Including implementation detail in a test name is fragile because implementation can change, making the test name confusing or incorrect. Changing an aspect of implementation should not cause us to rename our tests.
Additionally, CUBE is a client and we shouldn't identify, no matter how tenuously, that they are our client.
There was a problem hiding this comment.
Ok noted I shall use a different example for this.
There was a problem hiding this comment.
Yep that looks a lot better.
You still have an earlier mention of X-CUBE-ASSERT, so change that one also to match.
|
|
||
| ```test('should return a 400 when X-CUBE-ASSERT is Valid')``` | ||
|
|
||
| which uses single quotation marks ' as what you are passing into the name is a string. However, it can be difficult when you wish to pass in a variable to that same string, as variables are noted by single quotation marks also. |
There was a problem hiding this comment.
To be nitpicky, "as the name is a string" is more accurate.
Also I suggest putting that single-quote in backticks like `'` so it stands out in the text, like this:
single quotation marks
'as
There was a problem hiding this comment.
Ah sorry, my suggestion was unclear, i meant this:
as what you are passing into the name is a string
aswhat you are passing intothe name is a string
as the name is a string
_posts/2025-05-14-some-best-practices-for-accessible-automation.md
Outdated
Show resolved
Hide resolved
|
|
||
| ```test(`a test example to show a test name for a test under ticket RBP-000`, {tag: 'RBP-000'}, async()``` | ||
|
|
||
| Here you can see that the tag simply just needs to be placed at the end of the test name and is separated with a comma. For good practice this should be the ticket number that the work is being generated from. |
There was a problem hiding this comment.
Use "simply" or "just", not both.
It's not really as simple as separating with a comma, as you are needing to pass your (single) tag in as an object (hence the curly braces), as the second argument to the function.
Playwright docs state tags must begin with @ char:
https://playwright.dev/docs/test-annotations#tag-tests
In fact, playwright docs show using annotations for marking tests against an issue:
https://playwright.dev/docs/test-annotations#annotate-tests
There was a problem hiding this comment.
I have added the @ symbol as that is something which has been missed on the example. I do still think descibing it as simple is correct. Would you prefer it to say something along the lines of "tag object"
There was a problem hiding this comment.
The wording just felt odd on first read, as comma separation is just how we separate parameters in javascript, but we know what you mean and the more I read it the more it seems fine! So ignore me.
For good practice this should be the ticket number
Playwright suggestion is to use annotations for ticket number. Tags are used for filtering tests to run, or filtering test results, not for tracking / annotating tests with issue numbers, so I would argue this might not be Good Practice if you are following Playwright's own guides 😉
_posts/2025-05-14-some-best-practices-for-accessible-automation.md
Outdated
Show resolved
Hide resolved
Please add a direct link to your post here:
https://cfreestone-scottlogic.github.io/blog/2025/05/14/some-best-practices-for-accessible-automation.html
Have you (please tick each box to show completion):
npm installfollowed bynpx mdspell "**/{FILE_NAME}.md" --en-gb -a -n -x -tif that's your thing)Posts are reviewed / approved by your Regional Tech Lead.