Skip to content

Commit

Permalink
added new command channel,update doc
Browse files Browse the repository at this point in the history
Signed-off-by: benjamin.747 <benjamin.747@outlook.com>
  • Loading branch information
benjamin-747 committed Nov 21, 2022
1 parent 1b52647 commit 34fd076
Show file tree
Hide file tree
Showing 9 changed files with 398 additions and 305 deletions.
17 changes: 15 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ After download crates file, you can also sync rustup mirrors if that meets your

```bash
docker exec freighter bash -c 'freighter rustup download'
and
docker exec freighter bash -c 'freighter channel download'
```

After download all the files by using __freighter crates download__ and __freighter rustup download__, you can run upload command to upstream all your local files to s3, for example:
Expand All @@ -83,17 +85,28 @@ $ 0 2 * * * docker exec freighter bash -c 'freighter rustup download'

##### 1. Download the crates index with specify directory
```bash
$ freighter crates -c /mnt/volume_fra1_01 pull
$ freighter -c /mnt/volume_fra1_01 crates pull
```
##### 2. Download all crates file to local disk and then uoload to s3(you need to config s3cmd tools):
```bash
freighter crates download --init --upload
```
##### 3. Download crates file with multi-thread to specify directory:
```bash
freighter crates -t 128 -c /mnt/volume_fra1_01 download --init
freighter -c /mnt/volume_fra1_01 crates -t 128 download --init
```

##### 4. Download rustup init file:
```bash
freighter -c /mnt/volume_fra1_01 rustup -t 128 download
```

##### 5. Download rust toolchain files:
```bash
freighter -c /mnt/volume_fra1_01 channel -t 128 download
```


### How to contribute?

This project enforce the [DCO](https://developercertificate.org).
Expand Down
1 change: 1 addition & 0 deletions src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ USAGE:
Some common freight commands are (see all commands with --list):
crates Sync the index and crate files from the upstream to local, cloud or registry
rustup Sync the rustup and toolchain files from the upstream to local, cloud or registry
See 'freight help <command>' for more information on a specific command.\n"
)
Expand Down
144 changes: 144 additions & 0 deletions src/commands/channel.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
//! **channel** subcommand focus on download rust toolchains from upstream. The core
//! function implemented in the `src/crates/channel`.
//!
//!
//! **channel** subcommand provide major functions include:
//!
//! Arguments:
//! - __domain__: you can choose your own upstream by adding this arugment in command
//! - __download-threads__: specify the download threads to parallel download,
//! this param can be changed in the configuration file or pass it here
//! - __no-progressbar__: not implemented
//!
//! # download subcommand
//! - before each download, freighter will try to fetch the sha256 of the file and compare with local file if it exists
//! and will skip downloading if they are matching.
//!
//! - sync serveral rust toolchains version from upstream to local
//! - by default, this subcommand will fetch latest stable, beta, nightly and
//! the specified version in your toml config file: __rustup.sync_stable_versions__
//! - if you are using --version arguments in subcommand, freighter will only download the version you specified,
//! learn more about [rust release info](https://forge.rust-lang.org/index.html) here
//! - in the download process, freighter will first download the channel file, for example: channel-rust-1.29.toml
//!
//! Arguments:
//! - __clean__: clean history files read by config file after download successfully.
//! - __version__: only download the version you specified,
//! you can provide any version format supported by rust-org, such as stable, beta or nightly-2022-07-31.
//!
//! # upload subcommand
//! upload file to Object Storage Service compatible with [AWS S3](https://aws.amazon.com/s3/)
//! - Digitalocean Spaces
//! - Huawei Cloud OBS
//! - Alibaba Cloud OSS
//! - Tencent Cloud COS
//! - AWS S3
//! - minio
//! - Ceph
//!
//! Arguments:
//! - __bucket__: set the s3 bucket you want to upload files to, you must provide this param befor uplaod.
//!

use clap::{arg, ArgMatches};
use log::info;

use crate::cloud::s3::{S3cmd, CloudStorage};
use crate::commands::command_prelude::*;
use crate::config::Config;
use crate::crates::channel::{sync_rust_toolchain, ChannelOptions};
use crate::errors::FreightResult;

pub fn cli() -> clap::Command {
clap::Command::new("channel")
.subcommand(subcommand("download")
.arg(flag("clean", "clean up historical versions"))
.arg(arg!(-v --"version" <VALUE> "only download the version you specified"))
)
.subcommand(subcommand("upload")
.arg(
arg!(-b --"bucket" <VALUE> "set the s3 bucket you want to upload files to")
.required(true)
))
.subcommand_required(true)
.arg_required_else_help(true)
.about("Sync the Rust toolchain from the upstream to the local registry")
.arg(flag("no-progressbar", "Hide progressbar when start sync"))
.arg(arg!(-t --"download-threads" <VALUE> "specify the download thread count")
.value_parser(value_parser!(usize))
)
.arg(arg!(-d --"domain" <VALUE> "specify the source you want to sync from"))
.help_template(
"\
Sync the rust toolchian files from the upstream(static.rust-lang.org) to the local filesystem, other cloud
storage services, or other registries.
USAGE:
{usage}
OPTIONS:
{options}
EXAMPLES
1. Download toolchains from source domain(not necessary, default from static.rust-lang.org)
with 64 download threads and then clean historical files
freighter channel -t 64 -d https://www.example.com download --clean
2. Upload rust toolchains to s3 bucket:
freighter channel upload -b bucket-name
3. Download specify version:
freighter channel download -v nightly-2022-07-31
\n")
}

///
///
///
pub fn exec(config: &mut Config, args: &ArgMatches) -> FreightResult {
let work_dir = config
.work_dir
.as_ref()
.expect("something bad happened because work_dir is none");

let mut opts = ChannelOptions {
config: config.rustup.to_owned(),
dist_path: work_dir.join("freighter/dist"),
..Default::default()
};

if let Some(domain) = args.get_one::<String>("domain").cloned() {
opts.config.domain = domain;
}

if let Some(download_threads) = args.get_one::<usize>("download-threads").cloned() {
opts.config.download_threads = download_threads;
};

info!("ChannelOptions info : {:#?}", opts);

match args.subcommand() {
Some(("download", args)) => sync_rust_toolchain(&ChannelOptions {
clean: args.get_flag("clean"),
version: args.get_one::<String>("version").cloned(),
..opts
})?,
Some(("upload", args)) => {
let bucket_name = args.get_one::<String>("bucket").cloned().unwrap();
let s3cmd = S3cmd::default();
s3cmd.upload_folder(opts.dist_path.to_str().unwrap(), &bucket_name).unwrap();
}
Some((cmd, _)) => {
unreachable!("unexpected command {}", cmd)
}
None => {
unreachable!("unexpected command")
}
};

Ok(())
}
8 changes: 4 additions & 4 deletions src/commands/sync.rs → src/commands/crates.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
//! ```YAML
//! env:
//! URL_api: "https://crates.io/api/v1/crates"
//! URL_cdn: "https://static.crates.io/crates/{crate}/{crate}-{version}.crate"
//! URL_cdn: "https://static.crates.io/crates/sync{crate}/{crate}-{version}.crate"
//! URL_s3_primary: "https://crates-io.s3-us-west-1.amazonaws.com/crates/{crate}/{crate}-{version}.crate"
//! URL_s3_fallback: "https://crates-io-fallback.s3-eu-west-1.amazonaws.com/crates/{crate}/{crate}-{version}.crate"
//! ```
Expand Down Expand Up @@ -93,15 +93,15 @@ OPTIONS:
EXAMPLES
1. Sync the crates index with specify directory
freighter sync -c /mnt/volume_fra1_01 pull
freighter -c /mnt/volume_fra1_01 crates pull
2. Download all crates file and uoload:
freighter sync download --init --upload
freighter crates download --init --upload
3. Download crates file with multi-thread to specify directory:
freighter sync -t 512 -c /mnt/volume_fra1_01 download --init
freighter -c /mnt/volume_fra1_01 crates -t 512 download --init
\n")
}
Expand Down
11 changes: 7 additions & 4 deletions src/commands/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,9 @@ use crate::cli::App;
use crate::config::Config;
use crate::errors::FreightResult;

pub(crate) mod sync;
pub(crate) mod rustup;
pub mod crates;
pub mod rustup;
pub mod channel;
pub mod command_prelude;

/// The builtin function is the entry point of commands mod. Each subcommand is a
Expand All @@ -24,8 +25,9 @@ pub mod command_prelude;
///
pub fn builtin() -> Vec<App> {
vec![
sync::cli(),
crates::cli(),
rustup::cli(),
channel::cli(),
]
}

Expand All @@ -34,8 +36,9 @@ pub fn builtin() -> Vec<App> {
///
pub fn builtin_exec(cmd: &str) -> Option<fn(&mut Config, &ArgMatches) -> FreightResult> {
let f = match cmd {
"crates" => sync::exec,
"crates" => crates::exec,
"rustup" => rustup::exec,
"channel" => channel::exec,
_ => return None,
};

Expand Down
51 changes: 13 additions & 38 deletions src/commands/rustup.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,25 +8,12 @@
//! - __domain__: you can choose your own upstream by adding this arugment in command
//! - __download-threads__: specify the download threads to parallel download,
//! this param can be changed in the configuration file or pass it here
//! - __no-progressbar__: not implemented
//!
//! # download subcommand
//! - sync rustup init from upstream to local
//! - download subcommand will fetch only the latest version of init file, and this can't be changed by config.
//! - before each download, freighter will try to fetch the sha256 of the file and compare with local file if it exists
//! - download subcommand will fetch only the latest version of init file, and this can't be changed by config.
//! - before each download, freighter will try to fetch the sha256 of the file and compare with local file if it exists
//! and will skip downloading if they are matching.
//!
//! - sync serveral rust toolchains version from upstream to local
//! - by default, this subcommand will fetch latest stable, beta, nightly and
//! the specified version in your toml config file: __rustup.sync_stable_versions__
//! - if you are using --version arguments in subcommand, freighter will only download the version you specified,
//! learn more about [rust release info](https://forge.rust-lang.org/index.html) here
//! - in the download process, freighter will first download the channel file, for example: channel-rust-1.29.toml
//!
//! Arguments:
//! - __clean__: clean history files read by config file after download successfully.
//! - __version__: only download the version you specified,
//! you can provide any version format supported by rust-org, such as stable, beta or nightly-2022-07-31.
//!
//! # upload subcommand
//! upload file to Object Storage Service compatible with [AWS S3](https://aws.amazon.com/s3/)
Expand All @@ -45,17 +32,15 @@
use clap::{arg, ArgMatches};
use log::info;

use crate::cloud::s3::{S3cmd, CloudStorage};
use crate::commands::command_prelude::*;
use crate::config::Config;
use crate::crates::rustup::{sync_rustup, upload_to_s3, RustUpOptions};
use crate::crates::rustup::{sync_rustup_init, RustUpOptions};
use crate::errors::FreightResult;

pub fn cli() -> clap::Command {
clap::Command::new("rustup")
.subcommand(subcommand("download")
.arg(flag("clean", "clean up historical versions"))
.arg(arg!(-v --"version" <VALUE> "only download the version you specified"))
)
.subcommand(subcommand("download"))
.subcommand(subcommand("upload")
.arg(
arg!(-b --"bucket" <VALUE> "set the s3 bucket you want to upload files to")
Expand All @@ -64,14 +49,13 @@ pub fn cli() -> clap::Command {
.subcommand_required(true)
.arg_required_else_help(true)
.about("Sync the Rustup toolchain from the upstream to the local registry")
.arg(flag("no-progressbar", "Hide progressbar when start sync"))
.arg(arg!(-t --"download-threads" <VALUE> "specify the download thread count")
.value_parser(value_parser!(usize))
)
.arg(arg!(-d --"domain" <VALUE> "specify the source you want to sync from"))
.help_template(
"\
Sync the rust toolchian files from the upstream(static.rust-lang.org) to the local filesystem, other cloud
Sync the rustup init files from the upstream(static.rust-lang.org) to the local filesystem, other cloud
storage services, or other registries.
USAGE:
Expand All @@ -82,18 +66,14 @@ OPTIONS:
EXAMPLES
1. Download toolchains from source domain(not necessary, default from static.rust-lang.org)
with 64 download threads and then clean historical files
with 64 download threads
freighter rustup -t 64 -d https://www.example.com download --clean
freighter rustup -t 64 -d https://www.example.com download
2. Upload rustup init file and toolchains to s3 bucket:
2. Upload rustup init file to s3 bucket:
freighter rustup upload -b bucket-name
3. Download specify version:
freighter rustup download -v nightly-2022-07-31
\n")
}

Expand All @@ -108,9 +88,7 @@ pub fn exec(config: &mut Config, args: &ArgMatches) -> FreightResult {

let mut opts = RustUpOptions {
config: config.rustup.to_owned(),
no_progressbar: args.get_flag("no-progressbar"),
rustup_path: work_dir.join("freighter/rustup"),
dist_path: work_dir.join("freighter/dist"),
..Default::default()
};

Expand All @@ -125,14 +103,11 @@ pub fn exec(config: &mut Config, args: &ArgMatches) -> FreightResult {
info!("RustUpOptions info : {:#?}", opts);

match args.subcommand() {
Some(("download", args)) => sync_rustup(&RustUpOptions {
clean: args.get_flag("clean"),
version: args.get_one::<String>("version").cloned(),
..opts
})?,
Some(("download", _)) => sync_rustup_init(&opts)?,
Some(("upload", args)) => {
opts.bucket_name = args.get_one::<String>("bucket").cloned().unwrap();
upload_to_s3(&opts)?
let bucket_name = args.get_one::<String>("bucket").cloned().unwrap();
let s3cmd = S3cmd::default();
s3cmd.upload_folder(opts.rustup_path.to_str().unwrap(), &bucket_name).unwrap();
}
Some((cmd, _)) => {
unreachable!("unexpected command {}", cmd)
Expand Down
Loading

0 comments on commit 34fd076

Please sign in to comment.