Resubmission of templating PR (#1747) #3004

ehiggs · 2016-08-17T00:12:34Z

This is a manual rebase of the older #1747 PR which was basically unrebasable due to the time it's been dormant.. This implements templating for Cargo and is basically the work of Greg Chapple (@gchp).

I'd love for this feature to move forward since it's really tedious to create all the directories (e.g. src/bin) and copy paste docopt examples which I then edit. Then implement the main program to delegate to the subcommands in src/bin, etc.

rust-highfive · 2016-08-17T00:12:46Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

killercup · 2016-08-17T08:28:37Z

Nice! I would really love to have this. Copying the same README.md, LICENSE, travis, gitignore, editorconfig, etc. files into every repo can't be the way it's supposed to go.

gchp · 2016-08-17T08:31:30Z

Thanks for re-submitting this. I never got the time to come back to it unfortunately, but I think its very useful and would love to see it finished off and merged. Thanks for taking it on @ehiggs!

gchp · 2016-08-17T08:34:05Z

The biggest thing not addressed by this from the earlier PR was the use of Handlebars instead of Mustache. At the time the existing Handlebars implementations were a little unstable, but I imagine it's better now so should be easier to work in.

ehiggs · 2016-08-19T08:27:38Z

@gchp agreed. So the work is as follows:

Migrate to handlebars.
Remove caching mechanism. If users want a local version, they will have to manage the download and copying into ~/.cargo/template manually.
Tests
Documentation

Future work: cargo template suite of subcommands for managing templates.

alexcrichton · 2016-08-22T17:36:19Z

Cargo.lock

 "openssl-sys-extras 0.7.14 (registry+https://github.com/rust-lang/crates.io-index)",
 ]

 [[package]]
 name = "openssl-sys"
-version = "0.7.14"
+version = "0.7.17"


It looks like this may have run cargo update, but could this commit just add the dependencies it needs to?

alexcrichton · 2016-08-22T17:38:12Z

Thanks for the revival @ehiggs! Excited to see progress on this

alexcrichton · 2016-08-22T17:39:16Z

src/cargo/ops/cargo_new.rs

+            };
+            let template_dir = templates_dir.join(repo_name);
+
+            match Repository::clone(template, &*template_dir) {


Cargo typically issues a status message for operations that may take a long time, so perhaps this could indicate that a repository is being cloned?

I think this may also wish to use the helpers in git/utils.rs to ensure this handles pieces such as authentication, retries, etc.

ehiggs · 2016-08-30T23:39:08Z

I've done some work on this and rebased. I changed the way remote repos work by cloning them into a tempdir and using them as a template. The tempdir is then deleted when it goes out of scope. I do something similar with the default bin/lib templates. These are dumped to a tempdir and then used as a template.

I tried to use cargo::sources::git::utils::GitRemote and friends but when I try to clone the remote repository using checkout it seems to only copy the .git directory. I guess this is used to finely figure out which revisions are available or maybe I misunderstood how it's supposed to work since I thought there might be a nice single function to call to get all the work done.

In trying to get the git::utils working, I noticed that it only supports remote git repositories with Urls. This means remote repositories defined using ssh strings (e.g. git@github.com:ehiggs/rust-cmdline-skeleton.git) won't work.

I also made a skeleton repository to test this with. It used docopt and has a LICENSE file and everything to get started writing command line programs that use subcommands. It's basically @BurntSushi's xsv stripped down to... a skeleton 💀 (I keep calling it a skeleton since that's what it's called in adduser.

ehiggs · 2016-09-07T21:19:57Z

I've added some commits that address docs and fix the tests.

ehiggs · 2016-09-07T23:25:26Z

Travis failed due to a network issue. Appveyor failed on "override_cargo_home" but I don't have access to a Windows box to try and reproduce it.

alexcrichton · 2016-09-12T23:29:17Z

Thanks for the update @ehiggs! Some thoughts of mine:

The management of git templates still feels like we need to tweak the UI here. For example once it's cloned the first time it'll never get updated until it's removed? I'd expect that each time you create a project it'd update the git repo currently.
Storing custom templates in ~/.cargo/templates seems like it may be brittle wrt namespace collisions? It's also kinda weird to store user data in a dotfile location. I'd personally prefer if ~/.cargo/templates was entirely Cargo-controlled, for now. (but paths could be specified as template directories still)
Could a test be added for a git repository as well?
We may want to explore ~/.cargo/config configuration for default templates, I know I'd probably use it!

Also cc @rust-lang/tools, would be good to start hearing thoughts about this as well!

ehiggs · 2016-09-14T09:16:52Z

For example once it's cloned the first time it'll never get updated until it's removed?

There seems to be some confusion here. @gchp's original work cloned the remote repository into ~/.cargo/templates/<repo-name>. I removed that and merely clone the repo into a tempdir which is thrown away when the resulting Cargo project has been generated. So no local updating is required. Any work like that would be done in cargo template clone and/or cargo template pull, type commands which aren't in this PR.

So I guess the resulting work should be:

git tests using test/git.rs as an example.
Options to set the template dir in ~/.cargo/config
Options to set the default template in ~/.cargo/config

For the default template, I would keep --lib and --bin use the explicit defaults; but using no flag would use the default template instead of the --lib one.

brson · 2016-09-16T00:43:22Z

I like the feature and from reading the docs this seems pretty reasonable.

I've always envisioned project templates being part of the platform, so I would expect us to eventually distribute a bunch of them by default, and for IDEs to have options to instantiate them. This implementation doesn't provide for that yet since it only reads from .cargo and git repos. That's fine since we can improve the feature in the future, but there is one thing here I'm worried could prevent an important enhancement, and that's the way it gets templates from git repos.

As a step toward including templates in the Rust distribution by default I might expect and encourage somebody to maintain a project collecting templates - a single repo that contains nothing but templates. Thus people would naturally want to instantiate templates out of a subdirectory of a git repo, not from the top-level of the repo. This implementation seems to support one template per repo.

ehiggs · 2016-09-18T16:08:45Z

there is one thing here I'm worried could prevent an important enhancement, and that's the way it gets templates from git repos.

In writing the example repository I thought that it might be a pain the backside to maintain several repositories for each skeleton. I think the options would be:

Change the work so it uses a git repo and an optional second argument for a subdirectory. This is straightforward to implement.
Change the work to be single files and the template just works as a url to the file. This could be a .zip, a .tgz, or maybe even a .patch file which is applied to create the new files and then templated. I would not do something like a Dockerfile or spec since they runs arbitrary scripts and are expected to come from trusted/signed sources. This is more involved but it means that anyone who wants to serve up repos can just toss up a web page with a thin veneer over an object store and serve the templates.

ehiggs · 2016-09-19T19:31:00Z

After discussion with @brson on irc, we think this will work:

New lib: $ cargo new foo --lib
New bin: $ cargo new foo --bin

New local template:

$ grep template ~/.cargo/config
template-dir=~/myrusttemplates
default-template=mytemplate
$ cargo new foo --template mytemplate
$ # OR...
$ cargo new foo # no longer assumes `--lib`; uses `default-template setting`.

New template based on git repo:

$ cargo new foo --template-repo=https://gitlab.com/ehiggs/awesome-template

New template based on git repo where the repo is a collection of templates:

$ cargo new foo --template-repo=https://gitlab.com/ehiggs/awesome-templates --template=best-template

Probably new errors:

$ cargo new foo --template=lib # This will run a template called lib; not the builtin `--lib` version.
$ cargo new foo --template=bin # This will run a template called bin; not the builtin `--bin` version.

bors · 2016-09-28T01:58:34Z

☔ The latest upstream changes (presumably #3057) made this pull request unmergeable. Please resolve the merge conflicts.

alexcrichton · 2016-09-30T19:50:09Z

@ehiggs oh thanks for the previous clarifications! The strategy you and @brson came up with sounds reasonable to me, but I wonder if we could perhaps forgo the ~/.cargo configuration for now? Or is that required for particular portions?

If you want to rebase I can take another look as well.

ehiggs · 2016-10-07T23:13:41Z

I rebased and removed use of the config. So now there is --template and --template-repo.

The following is supported:

New default library: cargo new --lib foo

New default binary: cargo new --bin foo

New templated project based on remote repo:
cargo new --template-repo https://rustrepos.org/mytemplate foo

New templated project selecting a single template from a bunch in the repository:
cargo new --template-repo https://rustrepos.org/many-repos --template mytemplate foo

Local repo (untested ⚠️ ): cargo new --template-repo file://path/to/template foo

Local directory: cargo new --template-repo /path/to/template foo

Error (template with no associated repo): cargo new --template mytemplate foo

TheNeikos · 2016-10-10T11:45:07Z

src/cargo/ops/cargo_new.rs

+
+    let template_dir = match (opts.template_repo, opts.template) {
+        // given template is a remote git repository & needs to be cloned
+        // This will be cloned to .cargo/templates/<repo_name> where <repo_name>


This doesn't seem to be true anymore, I think that's a left over from previous iterations.

bors · 2016-10-10T20:32:59Z

☔ The latest upstream changes (presumably #3185) made this pull request unmergeable. Please resolve the merge conflicts.

ehiggs · 2016-10-14T12:03:16Z

I think travis had some network trouble:

Linux beta toolchain:

registry-b44aed2c1599b15a: /home/travis/.cargo/registry/src/github.com-88ac128001ac3a9a/libgit2-sys-0.4.5/libgit2/src/global.c:33: git__on_shutdown: Assertion `count <= 8 && count > 0' failed.
error: Process didn't exit successfully: `/home/travis/build/rust-lang/cargo/target/x86_64-unknown-linux-gnu/debug/registry-b44aed2c1599b15a` (signal: 6, SIGABRT: process abort signal)
To learn more, run the command again with --verbose.
make: *** [test-unit-x86_64-unknown-linux-gnu] Error 101

OS X stable toolchain:

find $(git ls-files) -type f \
        \( -perm -u+x -or -perm -g+x -or -perm -o+x \) \
        -not -name configure -not -name '*.sh' -not -name '*.rs' \
        -not -name '*.py' -not -wholename "*/rust-installer/*" | \
        grep '.*' \
        && exit 1 || exit 0
target/snapshot/bin/cargo test --target x86_64-apple-darwin -j1  
 Downloading hamcrest v0.1.1
 Downloading bufstream v0.1.2
 Downloading num v0.1.36
error: unable to get packages from source

alexcrichton · 2016-10-17T23:33:13Z

src/cargo/ops/cargo_new.rs

+            println!("Checking out repo to dir: {}", template_dir.path().display());
+
+            try!(config.shell().status("Cloning", repo_url));
+            match Repository::clone(repo_url, &*template_dir.path()) {


Could this use the support in the sources::git module? That way it'd handle cloning concerns like proxies and such.

alexcrichton · 2016-10-17T23:33:24Z

src/cargo/ops/cargo_new.rs

+            match Repository::clone(repo_url, &*template_dir.path()) {
+                Ok(_) => {},
+                Err(e) => {
+                    return Err(human(format!( 


Could this use chain_error and try!?

alexcrichton · 2016-10-17T23:33:40Z

src/cargo/ops/cargo_new.rs

+        // given template is a remote git repository & needs to be cloned
+        TemplateType::GitRepo(repo_url) => {
+            let template_dir = try!(TempDir::new(name));
+            println!("Checking out repo to dir: {}", template_dir.path().display());


Debug print?

alexcrichton · 2016-10-17T23:34:33Z

src/cargo/ops/cargo_new.rs

@@ -366,6 +461,10 @@ fn existing_vcs_repo(path: &Path, cwd: &Path) -> bool {
    GitRepo::discover(path, cwd).is_ok() || HgRepo::discover(path, cwd).is_ok()
 }

+fn file(p: &Path, contents: &[u8]) -> io::Result<()> {
+        try!(File::create(p)).write_all(contents)


extra indent?

I also think there may be a helper for this in cargo::util::paths? If not, could you add one there?

alexcrichton · 2016-10-17T23:35:35Z

src/cargo/ops/cargo_new.rs

-        }
+    // make sure that the template exists
+    if fs::metadata(&template_path).is_err() {
+        return Err(human(format!("Template `{}` not found", template_path.display())))


You can use the bail! macro here for a bit more ergonomic bailing out, and also stylistically Cargo error messages start with a lowercase letter.

alexcrichton · 2016-10-17T23:37:43Z

src/cargo/ops/cargo_new.rs

-    }
+
+        // create the new file & render the template to it
+        let mut dest_file = OpenOptions::new().write(true).create(true).open(&dest_path).unwrap();


This unwrap should be an error (I think there may also be a utility function for this)

alexcrichton · 2016-10-17T23:38:02Z

src/cargo/ops/cargo_new.rs

+fn walk_template_dir(dir: &Path, cb: &mut FnMut(DirEntry)) -> CargoResult<()> {
+    let attr = try!(fs::metadata(&dir));
+
+    let ignore_files = vec!(".gitignore");


Typically we use vec![] instead of vec!()

alexcrichton · 2016-10-17T23:38:53Z

src/cargo/ops/cargo_new.rs

+    let attr = try!(fs::metadata(&dir));
+
+    let ignore_files = vec!(".gitignore");
+    let ignore_types = vec!("png", "jpg", "gif");


This seems like a bit of an odd ignore list, shouldn't we try to walk over everything?

alexcrichton · 2016-10-17T23:39:00Z

src/cargo/ops/cargo_new.rs

+                    try!(walk_template_dir(&entry.path(), cb));
+                }
+            } else {
+                if let &Some(extension) = &entry.path().extension() {


Could these & sigils get removed?

alexcrichton · 2016-10-17T23:40:08Z

src/cargo/ops/cargo_new.rs

+            } else {
+                try!(create_lib_template(&temp_templates_path));
+            }
+            TemplateDirectory::Temp(temp_templates_dir)


If we are using a builtin template, could we avoid writing out to the filesystem? That shouldn't be strictly required, right?

Perhaps the template could get loaded into memory here in all cases?

ehiggs · 2016-10-26T13:59:40Z

I've fixed it so --bin and --lib no longer use the filesystem to write the template in an intermediate step. I've also pushed a lot of the template functionality to cargo/util/templates.rs.

bors · 2017-01-31T04:49:23Z

☔ The latest upstream changes (presumably #3618) made this pull request unmergeable. Please resolve the merge conflicts.

PR #3004 This is a resubmission of the PR #1747 (from scratch) which adds support for templating in Cargo. The templates are implemented using the handlebars crate (where the original PR used mustache). Examples: cargo new --template https://url/to/template somedir foo cargo new --template https://url/to/templates --template-subdir somedir foo cargo new --template ../path/to/template somedir foo

alexcrichton · 2017-01-31T22:26:51Z

Beta was branched let's merge!

@bors: r+

bors · 2017-01-31T22:26:51Z

📌 Commit 265452d has been approved by alexcrichton

bors · 2017-02-01T02:06:10Z

⌛ Testing commit 265452d with merge 5ef54e0...

ehiggs · 2017-02-01T08:27:39Z

Travis has had problems with the OS X queue. Hence the ❌

bors · 2017-02-01T09:10:03Z

💔 Test failed - status-travis

alexcrichton · 2017-02-01T17:16:23Z

@bors: retry

bors · 2017-02-01T22:22:24Z

⌛ Testing commit 265452d with merge f7fb965...

@gchp

Resubmission of templating PR (#1747) This is a manual rebase of the older #1747 PR which was basically unrebasable due to the time it's been dormant.. This implements templating for Cargo and is basically the work of Greg Chapple (@gchp). I'd love for this feature to move forward since it's really tedious to create all the directories (e.g. `src/bin`) and copy paste docopt examples which I then edit. Then implement the `main` program to delegate to the subcommands in `src/bin`, etc.

bors · 2017-02-02T00:09:01Z

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing f7fb965 to master...

alexcrichton · 2017-02-02T17:02:11Z

🎉

Thanks so much again @ehiggs for your patience!

ehiggs · 2017-02-02T21:34:42Z

Thank you for the patience. Now time to go bash out some templates. 😀
And a big thanks to @gchp for his original work!

gchp · 2017-02-06T18:54:55Z

@ehiggs I think it's safe to say you far out-did my original work 😄 great job!

…alexcrichton Restore the generated tests module created by `cargo new` Appears to have been removed unintentionally in #3004. Was just working on the book, ran `cargo new adder` with cargo-0.18.0-nightly (6f1b860 2017-02-11), and got this in `src/lib.rs`: ```rust #[test] fn it_works() { } ``` when I expected to get this: ```rust #[cfg(test)] mod tests { #[test] fn it_works() { } } ``` It looks like this was changed as part of #3004 ([removed](875a8ab#diff-149dd4362a3b0dc13b113762713119dfL477), [added](875a8ab#diff-149dd4362a3b0dc13b113762713119dfR678)), I'm assuming unintentionally? The regression has not yet hit the beta channel; `cargo-0.17.0-nightly (0bb8047 2017-02-06)` generates the src/lib.rs I expect.

WiSaGaN · 2017-08-11T03:37:10Z

A note that this pull-request has been reverted in #3878 in waiting for a RFC.

ehiggs · 2017-08-11T08:56:47Z

Indeed, ongoing discussion is here. It seems there is no consensus yet.

rust-highfive assigned alexcrichton Aug 17, 2016

alexcrichton reviewed Aug 22, 2016
View reviewed changes

ehiggs closed this Oct 7, 2016

ehiggs reopened this Oct 7, 2016

TheNeikos reviewed Oct 10, 2016

View reviewed changes

alexcrichton reviewed Oct 17, 2016

View reviewed changes

Ewan Higgs and others added 3 commits January 31, 2017 10:23

Upgrade handlebars to 0.25.

0aad658

Remove unused 'file' function.

265452d

bors merged commit 265452d into rust-lang:master Feb 2, 2017

bors mentioned this pull request Feb 2, 2017

Add "Benchmarks" section to Guide #3472

Closed

vi mentioned this pull request Feb 17, 2017

cargo init no longer handles pre-existing source files nicely #3722

Closed

alexcrichton mentioned this pull request Mar 3, 2017

List of default dependencies for all projects #3793

Closed

carols10cents mentioned this pull request Mar 4, 2017

Restore the generated tests module created by cargo new #3799

Merged

alexcrichton mentioned this pull request Mar 10, 2017

couldn't read src/main.rs in reustmann-0.1.2, Rust 1.17 rust-lang/rust#40185

Closed

ssokolow mentioned this pull request Mar 13, 2017

Enable syntax highlighting for emacs ssokolow/rust-cli-boilerplate#3

Merged

carols10cents mentioned this pull request Jan 8, 2018

cargo new: create a README #3506

Closed

carols10cents mentioned this pull request Mar 8, 2018

Feature request: Cargo templates #5151

Open

ehuss added this to the 1.17.0 milestone Feb 6, 2022

Resubmission of templating PR (#1747) #3004

Resubmission of templating PR (#1747) #3004

Conversation

ehiggs commented Aug 17, 2016

rust-highfive commented Aug 17, 2016

killercup commented Aug 17, 2016

gchp commented Aug 17, 2016

gchp commented Aug 17, 2016

ehiggs commented Aug 19, 2016 • edited Loading

Choose a reason for hiding this comment

alexcrichton commented Aug 22, 2016

Choose a reason for hiding this comment

ehiggs commented Aug 30, 2016 • edited Loading

ehiggs commented Sep 7, 2016

ehiggs commented Sep 7, 2016

alexcrichton commented Sep 12, 2016

ehiggs commented Sep 14, 2016 • edited Loading

brson commented Sep 16, 2016

ehiggs commented Sep 18, 2016

ehiggs commented Sep 19, 2016

bors commented Sep 28, 2016

alexcrichton commented Sep 30, 2016

ehiggs commented Oct 7, 2016

Choose a reason for hiding this comment

bors commented Oct 10, 2016

ehiggs commented Oct 14, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ehiggs commented Oct 26, 2016

bors commented Jan 31, 2017

alexcrichton commented Jan 31, 2017

bors commented Jan 31, 2017

bors commented Feb 1, 2017

ehiggs commented Feb 1, 2017

bors commented Feb 1, 2017

alexcrichton commented Feb 1, 2017

bors commented Feb 1, 2017

bors commented Feb 2, 2017

alexcrichton commented Feb 2, 2017

ehiggs commented Feb 2, 2017 • edited Loading

gchp commented Feb 6, 2017

WiSaGaN commented Aug 11, 2017

ehiggs commented Aug 11, 2017

ehiggs commented Aug 19, 2016 •

edited

Loading

ehiggs commented Aug 30, 2016 •

edited

Loading

ehiggs commented Sep 14, 2016 •

edited

Loading

ehiggs commented Oct 14, 2016 •

edited

Loading

ehiggs commented Feb 2, 2017 •

edited

Loading