Add import command to import files from non-bender repositories (#99)

* Add vendor config to Manifest
* Change git linking to not require session
* Add `import` command

CHANGELOG.md

@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 ### Added
 - Add clippy to github CI and fix noted issues
+- Add `import` command to import files from non-bender repositories
 
 ### Changed
 - Update `tokio` dependency, update to `futures=0.3`, rewrite for `async/await`. Bumps minimum rust to 1.57

Cargo.toml

@@ -28,3 +28,5 @@ itertools = "0.10"
 atty = "0.2"
 tabwriter = "1.2.1"
 indexmap = "1.8.1"
+tempdir = "0.3.7"
+glob = "0.3"

README.md

@@ -175,6 +175,26 @@ workspace:
 # Optional. Only available in dependent packages.
 plugins:
   hello: scripts/hello.sh
+
+# List of imported files from external repositories not supporting bender. Optional
+external_import:
+    # package name
+  - name: lowrisc_opentitan
+    # target directory
+    target_dir: vendor/lowrisc_opentitan
+    # upstream dependency (i.e. git repository similar to dependencies)
+    upstream: { git: "https://github.com/lowRISC/opentitan.git", rev: "47a0f4798febd9e53dd131ef8c8c2b0255d8c139" }
+    # paths to exclude from upstream dependency
+    exclude_from_upstream:
+      - "ci/*"
+    # directory containing patch files
+    patch_dir: "vendor/patches"
+    # file mapping from remote repository to local repository, with optional patch_dir containing patches
+    mapping: 
+      - {from: 'hw/ip/prim/rtl/prim_subreg.sv', to: 'src/prim_subreg.sv' }
+      - {from: 'hw/ip/prim/rtl/prim_subreg_arb.sv', to: 'src/prim_subreg_arb.sv' }
+      - {from: 'hw/ip/prim/rtl/prim_subreg_ext.sv', to: 'src/prim_subreg_ext.sv', patch_dir: 'lowrisc_opentitan' }
+      - {from: 'hw/ip/prim/rtl/prim_subreg_shadow.sv', to: 'src/prim_subreg_shadow.sv' }
 ```
 
 [Relevant code](https://github.com/pulp-platform/bender/blob/master/src/config.rs)
@@ -291,6 +311,11 @@ Additionally, we suggest to use the following targets to identify source code an
 
 [Relevant code](https://github.com/pulp-platform/bender/blob/master/src/target.rs)
 
+### Vendor
+
+Section to list files and directories copied and patched within this repository from external repositories not supporting bender.
+To update, see below `vendor` command.
+
 
 ## Configuration Format (`bender.yml`, `Bender.local`)
 
@@ -429,6 +454,16 @@ The `bender parents <PKG>` command lists all packages calling the `PKG` package.
 
 This command will ensure all dependencies are downloaded from remote repositories. This is usually automatically executed by other commands, such as `sources` and `script`.
 
+### `import` --- Copy files from dependencies that do not support bender
+
+This command will update the dependencies listed in the `external_import` section of the `Bender.yml` file, fetching the files from the remote repositories and applying the necessary patch files.
+This command will print a diff to the patched remote repository.
+If the `--refetch` argument is passed, it will refetch the upstream and apply patched.
+If the `--gen_patch` argument is passed, it will generate additional new patches from the current diff.
+If the `-n/--no_patch` argument is passed, any application of current patches will be suppressed, with `--gen_patch` any existing patch files will be deleted and a new patch created.
+Please make sure you manage the includes and sources required for these files separately, as this command only fetches the files and patches them.
+This is in part based on [lowRISC's `vendor.py` script](https://github.com/lowRISC/opentitan/blob/master/util/vendor.py).
+
 [aur-bender]: https://aur.archlinux.org/packages/bender
 [releases]: https://github.com/pulp-platform/bender/releases
 [rust-installation]: https://doc.rust-lang.org/book/ch01-01-installation.html

src/cli.rs

@@ -54,7 +54,8 @@ pub fn main() -> Result<()> {
         .subcommand(cmd::sources::new())
         .subcommand(cmd::config::new())
         .subcommand(cmd::script::new())
-        .subcommand(cmd::checkout::new());
+        .subcommand(cmd::checkout::new())
+        .subcommand(cmd::import::new());
 
     // Add the `--debug` option in debug builds.
     let app = if cfg!(debug_assertions) {
@@ -234,6 +235,7 @@ pub fn main() -> Result<()> {
         Some(("script", matches)) => cmd::script::run(&sess, matches),
         Some(("checkout", matches)) => cmd::checkout::run(&sess, matches),
         Some(("update", _)) => Ok(()),
+        Some(("import", matches)) => cmd::import::run(&sess, matches),
         Some((plugin, matches)) => execute_plugin(&sess, plugin, matches.values_of_os("")),
         _ => Ok(()),
     }