Skip to content

Add custom flags to config #1670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 7, 2022
Merged

Add custom flags to config #1670

merged 3 commits into from
Feb 7, 2022

Conversation

SleeplessByte
Copy link
Member

@SleeplessByte SleeplessByte commented Feb 6, 2022

Supports exercism/javascript-test-runner#79.

This adds hand-written flags to the .meta/config.json files to be consumed by our tooling.

"version.tests.compatibility": "jest-27"

This doesn't do anything by itself, but jest 27 is technically a breaking change. Adding this to our config files allows us to:

  • separate exercises that have been updated vs. those that are on pre-27 versions
  • give special error messages if something timeout related happens. Why? Jest 27 imposes a standard 5s timeout on each individual test.
  • track unupdated exercises the next time jest is breaking
"flag.tests.task-per-describe": true

This should generally be true for all concept exercises and false for all practice exercises. When true, the test-runner can calculate the task_id based on the describe in the test files. This PR also contains updates to the spec files to match this newly imposed format.

"flag.tests.may-run-long": false

When we see students complain about timeouts on naive implementations, we should turn this to true. Doesn't do something right now, but if we see it a lot, we can use it to actually impose a shorter timeout (in the runner) and then show guided help.

"flag.tests.includes-optional": false

When true, skipped tests with test.skip are present and optional. This allows us to:

  • enforce in the test-runner that there are only skipped tests if this is true
  • add a message in the analyzer that there are optional tests that the student can run via the CLI. We can add an informational message with data how to do that, even if they did run this via the CLI.

@SleeplessByte SleeplessByte requested a review from junedev February 6, 2022 22:12
@github-actions

This comment was marked as resolved.

Copy link
Member

@junedev junedev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skimmed this. LGTM

@junedev
Copy link
Member

junedev commented Feb 7, 2022

Would be good to add the things you explained in the description here in the test runner readme if you didn't do that already.

@SleeplessByte
Copy link
Member Author

Yeah, I want to add it there, and also in our CONTRIBUTING guide, right?

@SleeplessByte SleeplessByte merged commit ed5f21b into main Feb 7, 2022
@SleeplessByte SleeplessByte deleted the feature/tooling-flags branch February 7, 2022 16:56
@junedev
Copy link
Member

junedev commented Feb 7, 2022

🤷‍♀️ I don't have an overview yet what documentation should go where ... getting a better grip around the docs is still on my todo list. So I'm fine as long as you add it somewhere.

@SleeplessByte
Copy link
Member Author

I am kinda leaning towards adding everything to the exercism.org/ javascript docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants