Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to test doc strings #11120

Merged
merged 6 commits into from
Dec 23, 2013
Merged

Conversation

alexcrichton
Copy link
Member

This commit adds a --test flag to rustdoc to expose the ability to test code examples in doc strings. This work by using sundown's lang attribute to figure out how a code block should be tested. The format for this is:

1. ```rust
2. ```rust,ignore
3. ```rust,notest
4. ```rust,should_fail

Where rust means that rustdoc will attempt to test is, ignore means that it will not execute the test but it will compile it, notest means that rustdoc completely ignores it, and should_fail means that the test should fail. This commit also leverages extra::test for the testing harness in order to allow parallelization and whatnot.

I have fixed all existing code examples in libstd and libextra, but I have not made a pass through the crates in order to change code blocks to testable code blocks.

It may also be a questionable decision to require opting-in to a testable code block.

Finally, I kept our sugar in the doc suite to omit lines starting with # in documentation but still process them during tests.

Closes #2925

@sfackler
Copy link
Member

Could you update the rustdoc tutorial to mention this while you're at it? It seems pretty threadbare at the moment, but it'd be nice to keep it from falling further behind.

@alexcrichton
Copy link
Member Author

I have added some documentation in the rustdoc tutorial

@huonw
Copy link
Member

huonw commented Dec 23, 2013

Yay!

The doc target is (for example) make check-stage2-doc-std, right? Does this obey NO_REBUILD?

(Also, it'd be worth adding to this page when this lands.)

@alexcrichton
Copy link
Member Author

Yes, the target will be make check-stage2-doc-std, but it won't quite obey NO_REBUILD because it requires rustdoc to be built (which isn't gated on the NO_REBUILD env var. It's super easy to run though in that all this does is:

./$target/bin/rustdoc --test src/libstd/lib.rs

I'll definitely add it to the testing wiki! I'll reference the rustdoc documentation I wrote for that.

@huonw
Copy link
Member

huonw commented Dec 23, 2013

Couldn't NO_REBUILD also gate all the dependencies of this doc target? (Like it just gates all the dependencies of check-stage2-std.)

I.e. if someone has done a full make recently, and are just fiddling with std, then the rustdoc they've already built will be new enough.

@brson
Copy link
Contributor

brson commented Dec 23, 2013

It does seem like we may want to assume that all code blocks are rust and testable unless otherwise specified.

@brson
Copy link
Contributor

brson commented Dec 23, 2013

Very cool.

This adds support for the `--test` flag to rustdoc which will parse a crate,
extract all code examples in doc comments, and then run each test in the
extra::test driver.
Don't run doc tests during make check-fast because it involves spawning lots of
processes.
bors added a commit that referenced this pull request Dec 23, 2013
This commit adds a `--test` flag to rustdoc to expose the ability to test code examples in doc strings. This work by using sundown's `lang` attribute to figure out how a code block should be tested. The format for this is:

```
1. ```rust
2. ```rust,ignore
3. ```rust,notest
4. ```rust,should_fail
```

Where `rust` means that rustdoc will attempt to test is, `ignore` means that it will not execute the test but it will compile it, `notest` means that rustdoc completely ignores it, and `should_fail` means that the test should fail. This commit also leverages `extra::test` for the testing harness in order to allow parallelization and whatnot.

I have fixed all existing code examples in libstd and libextra, but I have not made a pass through the crates in order to change code blocks to testable code blocks.

It may also be a questionable decision to require opting-in to a testable code block.

Finally, I kept our sugar in the doc suite to omit lines starting with `#` in documentation but still process them during tests.

Closes #2925
@bors bors closed this Dec 23, 2013
@bors bors merged commit f9b231c into rust-lang:master Dec 23, 2013
@alexcrichton alexcrichton deleted the rustdoc-test branch December 23, 2013 18:45
@emberian
Copy link
Member

!!!! \o/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

generate tests from code example docstrings (to fight bitrot)
6 participants