New weight meter PR (open-web3-stack#478)

* init weight_meter * update meter static var * update workspace * fmt * clippy allow unused imports * default std * update cargo * remove std * revert until fix no_std issue * improvements * fmt * update argument extract * package name with orml prefix * update meter to be Mutex * use mutex only in std * update README * update README * update property name * separate method injector * docs * fmt * clippy * Orml bencher (open-web3-stack#452) * Use MultiLocation as xtokens transfer dest type. (open-web3-stack#396) * Use MultiLocation as xtokens transfer dest type. * Make clippy happy. * Use xcm-handler to execute XCM locally (open-web3-stack#401) * Use cumulus xcm-handler to execute XCM locally. * Add docstring for xtokens config. * Replace XcmError::Undefined usage. (open-web3-stack#402) * Replace XcmError::Undefined usage. * make clippy happy * Bump and unify serde versions. (open-web3-stack#403) * xtokens and xcm-support documentations (open-web3-stack#404) * Add xtokens module docstring. * Add xtokens readme. * Add documentations for xcm-support. * Add xtokens and xcm-support entries in main readme. * Add unit tests for xcm-support. (open-web3-stack#405) * Added Minterest to the list of users. (open-web3-stack#406) * update step guide link * Handle unknown assets in TransactAsset impl (open-web3-stack#409) * Handle unknown assets in TransactAsset impl. * More documentations. * Clean code. * Renaming. * Should try to deposit known asset first. * Return error if no UnknownAsset impl. * Make clippy happy. * Fix description and repo link. (open-web3-stack#410) * Unknown tokens pallet (open-web3-stack#411) * Impl unknown tokens pallet. * Fix workspace pallet path. * Make clippy happy. * Clippy, be happy. * Unit tests. * Remove nonces from oracle pallet. (open-web3-stack#413) * refactor rewards (open-web3-stack#412) * Bump rococo v1 dependencies (open-web3-stack#418) * Fix mocks. * Replace deprecated. * Update orml-unknown-tokens unit tests. (open-web3-stack#421) * add build-script-utils from Substrate (open-web3-stack#422) * Update README.md (open-web3-stack#420) * Update README.md * Update README.md * Bump impl-trait-for-tuples to 0.2.1 (open-web3-stack#424) * update Cargo.toml (open-web3-stack#429) * bencher init commit * split into files * clean deps * add docs * use frame_benchmarking apis * add macro bencher_use to export stuff for bench_runner * * generate weights file * refactor * improvements * CLI Options (open-web3-stack#446) * Added command line options for output, headers, and templates * Fixed options to conform to standard cli * Added weight-gen cli * fixed dependencies * Replaced unwraps with expects (open-web3-stack#449) * Orml bencher (open-web3-stack#451) * add Handler (open-web3-stack#431) * remove disable-tokens-by-owner (open-web3-stack#434) * Cross-chain transfer rework (open-web3-stack#432) * Reserve chain trait. * Rework cross-chain transfer. * Remove relay chain balance convert. * Add 'Parse' trait. * Change transfer_multiasset fn signature. * Add transfer dispatchable call. * Update doc. * Use xcm-simulator to mock network. * Send relay chain asset to sibling unit test. * Move location traits into orml-traits. * Add MultiNativeAsset filter for is reserve check. * More unit tests. * Failing edge case unit tests. * Handle zero amount asset case. * Fix mocks. * Renaming. * Update currency adapter to work with new xtokens impl (open-web3-stack#436) * Xcm support implementations rework. * Update xtokens mock. * Use CurrencyId convert. (open-web3-stack#437) * Use CurrencyId convert. * Apply review suggestions. * Update xtokens docs. (open-web3-stack#438) * Update xtokens docs. * Fix typo. * Update imbalances impl. * Don't deposit failure event in orml-unknown-tokens. (open-web3-stack#440) * Don't deposit failure event in orml-unknown-tokens. * Patch substrate/polkadot/cumulus. * Fix patch. * Update README.md (open-web3-stack#441) Include Zeitgeist into "Projects using ORML" section * Add PoV size in benchmarking. (open-web3-stack#442) * Bump cumulus ref in cargo patch. (open-web3-stack#443) * fix missing features (open-web3-stack#444) * fix missing features * test with benchmarks * update auction weight (open-web3-stack#445) * Bump dependencies. (open-web3-stack#448) * Replaced unwraps with expects Co-authored-by: Xiliang Chen <xlchen1291@gmail.com> Co-authored-by: Shaun Wang <spxwang@gmail.com> Co-authored-by: Harald Heckmann <harald.heckmann93@web.de> Co-authored-by: wangjj9219 <183318287@qq.com> Co-authored-by: Shaun Wang <spxwang@gmail.com> Co-authored-by: dzianis.ramanouski <DenisRomanovsky@users.noreply.github.com> Co-authored-by: Bette <42193328+bette7@users.noreply.github.com> Co-authored-by: wangjj9219 <183318287@qq.com> Co-authored-by: Xiliang Chen <xlchen1291@gmail.com> Co-authored-by: transxask <68648225+transxask@users.noreply.github.com> Co-authored-by: Aaro Perämaa <aaro.peramaa@gmail.com> Co-authored-by: Ermal Kaleci <ermalkaleci@gmail.com> Co-authored-by: Harald Heckmann <harald.heckmann93@web.de> * fix bencher dependencies * Weight meter (open-web3-stack#459) * Added argument to BenchmarkingState * fix * Updated frame-support version (open-web3-stack#461) * clippy * fmt * fix * switched to thread_local * Added checked add * corrected versions * weight_meter tests * Removed starting weight * Removed method_benchmark/updated inner docs * clippy and fmt * changed start_with to start * Changed start_with to start in macro * cleanup * refactor * fix clippy Co-authored-by: brettkolodny <brettkolodny@gmail.com> Co-authored-by: Shaun Wang <spxwang@gmail.com> Co-authored-by: dzianis.ramanouski <DenisRomanovsky@users.noreply.github.com> Co-authored-by: Bette <42193328+bette7@users.noreply.github.com> Co-authored-by: wangjj9219 <183318287@qq.com> Co-authored-by: Xiliang Chen <xlchen1291@gmail.com> Co-authored-by: transxask <68648225+transxask@users.noreply.github.com> Co-authored-by: Aaro Perämaa <aaro.peramaa@gmail.com> Co-authored-by: Harald Heckmann <harald.heckmann93@web.de>
interlay · May 17, 2021 · cda5f64 · cda5f64
1 parent 8825fe4
commit cda5f64
Show file tree

Hide file tree

Showing 18 changed files with 1,086 additions and 0 deletions.
diff --git a/Cargo.dev.toml b/Cargo.dev.toml
@@ -4,6 +4,7 @@ cargo-features = ["resolver"]
 members = [
 	"auction",
 	"authority",
+	"bencher",
 	"benchmarking",
 	"currencies",
 	"gradually-update",
@@ -20,6 +21,8 @@ members = [
 	# "xcm-support",
 	# "unknown-tokens",
 	"build-script-utils",
+	"weight-gen",
+	"weight-meter",
 ]
 resolver = "2"
 
@@ -67,6 +70,7 @@ sp-authority-discovery = { git = "https://github.com/paritytech//substrate", rev
 sc-executor-common = { git = "https://github.com/paritytech//substrate", rev = "3f110196163b5ec03bac5ee188d60bedf3ebd91d" }
 sc-executor-wasmi = { git = "https://github.com/paritytech//substrate", rev = "3f110196163b5ec03bac5ee188d60bedf3ebd91d" }
 sc-executor = { git = "https://github.com/paritytech//substrate", rev = "3f110196163b5ec03bac5ee188d60bedf3ebd91d" }
+sc-client-db = { git = "https://github.com/paritytech//substrate", rev = "3f110196163b5ec03bac5ee188d60bedf3ebd91d" }
 sc-client-api = { git = "https://github.com/paritytech//substrate", rev = "3f110196163b5ec03bac5ee188d60bedf3ebd91d" }
 sp-tasks = { git = "https://github.com/paritytech//substrate", rev = "3f110196163b5ec03bac5ee188d60bedf3ebd91d" }
 sp-authorship = { git = "https://github.com/paritytech//substrate", rev = "3f110196163b5ec03bac5ee188d60bedf3ebd91d" }

diff --git a/bencher/Cargo.toml b/bencher/Cargo.toml
@@ -0,0 +1,39 @@
+[package]
+name = "orml-bencher"
+description = "Provide macro to benchmark pallets."
+repository = "https://github.com/open-web3-stack/open-runtime-module-library/tree/master/bencher"
+license = "Apache-2.0"
+version = "0.4.1-dev"
+authors = ["Laminar Developers <hello@laminar.one>"]
+edition = "2018"
+
+[dependencies]
+linregress = { version = "0.4.0", optional = true }
+serde = { version = "1.0.119", optional = true, features = ['derive'] }
+serde_json = {version = "1.0.64", optional = true }
+codec = { package = "parity-scale-codec", version = "2.0.0", features = ["derive"], default-features = false }
+sp-core = { git = "https://github.com/paritytech/substrate", branch = "rococo-v1", default-features = false }
+sp-std = { git = "https://github.com/paritytech/substrate", branch = "rococo-v1", default-features = false }
+sp-io = { git = "https://github.com/paritytech/substrate", branch = "rococo-v1", default-features = false }
+sp-runtime-interface = { git = "https://github.com/paritytech/substrate", branch = "rococo-v1", default-features = false }
+sp-state-machine = { git = "https://github.com/paritytech/substrate", branch = "rococo-v1", default-features = false, optional = true }
+sc-executor = { git = "https://github.com/paritytech/substrate", branch = "rococo-v1", default-features = false, features = ["wasmtime"], optional = true }
+sc-client-db = { git = "https://github.com/paritytech/substrate", branch = "rococo-v1", default-features = false, features = ["with-kvdb-rocksdb"], optional = true }
+frame-benchmarking = { git = "https://github.com/paritytech/substrate", branch = "rococo-v1", default-features = false }
+
+[features]
+default = ["std"]
+std = [
+    "linregress",
+    "serde/std",
+    "serde_json/std",
+    "codec/std",
+    "sp-core/std",
+    "sp-std/std",
+    "sp-io/std",
+    "sp-runtime-interface/std",
+    "sp-state-machine/std",
+    "sc-executor/std",
+    "sc-client-db",
+    "frame-benchmarking/std",
+]
diff --git a/bencher/src/bench_runner.rs b/bencher/src/bench_runner.rs
@@ -0,0 +1,39 @@
+use frame_benchmarking::{
+	benchmarking,
+	frame_support::sp_runtime::traits::{Block, NumberFor},
+};
+use sc_client_db::BenchmarkingState;
+use sc_executor::{sp_wasm_interface::HostFunctions, WasmExecutionMethod, WasmExecutor};
+use sp_core::traits::{CallInWasm, MissingHostFunctions};
+use sp_io::SubstrateHostFunctions;
+use sp_state_machine::{Ext, OverlayedChanges, StorageTransactionCache};
+
+/// Run benches
+pub fn run<B: Block>(wasm_code: Vec<u8>) -> Vec<u8> {
+	let mut overlay = OverlayedChanges::default();
+	let mut cache = StorageTransactionCache::default();
+	let state = BenchmarkingState::<B>::new(Default::default(), Default::default(), false).unwrap();
+	let mut ext = Ext::<_, NumberFor<B>, _>::new(&mut overlay, &mut cache, &state, None, None);
+
+	let mut host_functions = benchmarking::HostFunctions::host_functions();
+	host_functions.append(&mut SubstrateHostFunctions::host_functions());
+
+	let executor = WasmExecutor::new(
+		WasmExecutionMethod::Compiled,
+		Default::default(),
+		host_functions,
+		1,
+		None,
+	);
+
+	executor
+		.call_in_wasm(
+			&wasm_code[..],
+			None,
+			"run_benches",
+			&[],
+			&mut ext,
+			MissingHostFunctions::Disallow,
+		)
+		.unwrap()
+}
diff --git a/bencher/src/handler.rs b/bencher/src/handler.rs
@@ -0,0 +1,54 @@
+use crate::BenchResult;
+use codec::Decode;
+use linregress::{FormulaRegressionBuilder, RegressionDataBuilder};
+use serde::{Deserialize, Serialize};
+use std::io::Write;
+
+#[derive(Serialize, Deserialize, Default, Debug, Clone)]
+struct BenchData {
+	pub name: String,
+	pub base_weight: u64,
+	pub base_reads: u32,
+	pub base_writes: u32,
+}
+
+/// Handle bench results
+pub fn handle(output: Vec<u8>) {
+	let results = <Vec<BenchResult> as Decode>::decode(&mut &output[..]).unwrap();
+	let data: Vec<BenchData> = results
+		.into_iter()
+		.map(|result| {
+			let name = String::from_utf8_lossy(&result.method).to_string();
+
+			eprintln!("{:#?}", result);
+
+			let y: Vec<f64> = result.elapses.into_iter().map(|x| x as f64).collect();
+			let x: Vec<f64> = (0..y.len()).into_iter().map(|x| x as f64).collect();
+			let data = vec![("Y", y), ("X", x)];
+			let data = RegressionDataBuilder::new().build_from(data).unwrap();
+			let formula = "Y ~ X";
+
+			let model = FormulaRegressionBuilder::new()
+				.data(&data)
+				.formula(formula)
+				.fit()
+				.unwrap();
+
+			BenchData {
+				name,
+				base_weight: model.parameters.intercept_value as u64 * 1_000,
+				base_reads: result.reads,
+				base_writes: result.writes,
+			}
+		})
+		.collect();
+
+	if let Ok(json) = serde_json::to_string(&data) {
+		let stdout = ::std::io::stdout();
+		let mut handle = stdout.lock();
+
+		handle.write_all(&json.as_bytes()).unwrap();
+	} else {
+		eprintln!("Could not write benchdata to JSON");
+	}
+}
diff --git a/bencher/src/lib.rs b/bencher/src/lib.rs
@@ -0,0 +1,28 @@
+#![cfg_attr(not(feature = "std"), no_std)]
+
+#[doc(hidden)]
+pub extern crate frame_benchmarking;
+#[doc(hidden)]
+pub extern crate sp_core;
+#[doc(hidden)]
+pub extern crate sp_std;
+
+use codec::{Decode, Encode};
+use sp_std::prelude::Vec;
+
+#[derive(Encode, Decode, Default, Clone, PartialEq, Debug)]
+pub struct BenchResult {
+	pub method: Vec<u8>,
+	pub elapses: Vec<u128>,
+	pub reads: u32,
+	pub repeat_reads: u32,
+	pub writes: u32,
+	pub repeat_writes: u32,
+}
+
+mod macros;
+
+#[cfg(feature = "std")]
+pub mod bench_runner;
+#[cfg(feature = "std")]
+pub mod handler;
diff --git a/bencher/src/macros.rs b/bencher/src/macros.rs
@@ -0,0 +1,160 @@
+/// Run benches in WASM environment.
+///
+/// Configure your module to build the mock runtime into wasm code.
+/// Create a `build.rs` like you do with your runtime.
+/// ```.ignore
+/// use substrate_wasm_builder::WasmBuilder;
+/// fn main() {
+///     WasmBuilder::new()
+///         .with_current_project()
+///         .export_heap_base()
+///         .import_memory()
+///         .build()
+/// }
+/// ```
+///
+/// Update mock runtime to be build into wasm code.
+/// ```.ignore
+/// #![cfg_attr(not(feature = "std"), no_std)]
+///
+/// #[cfg(feature = "std")]
+/// include!(concat!(env!("OUT_DIR"), "/wasm_binary.rs"));
+///
+/// #[cfg(feature = "std")]
+/// pub fn wasm_binary_unwrap() -> &'static [u8] { WASM_BINARY.unwrap() }
+/// ..
+/// ```
+///
+/// Create a file `bench_runner.rs` with following code:
+///  ```.ignore
+/// orml_bencher::run_benches!(my_module::benches);
+/// ```
+/// 
+/// Update Cargo.toml by adding:
+/// ```toml
+/// ..
+/// [package]
+/// name = "my-module"
+/// ..
+/// build = 'build.rs'
+///
+/// [build-dependencies]
+/// substrate-wasm-builder = '4.0.0'
+///
+/// [[bench]]
+/// name = 'benches'
+/// harness = false
+/// path = 'bench_runner.rs'
+/// required-features = ['bench']
+///
+/// [features]
+/// bench = []
+/// ..
+/// ```
+/// 
+/// Run bench with features bench: `cargo bench --features=bench`
+#[cfg(feature = "std")]
+#[macro_export]
+macro_rules! run_benches {
+	($benches:path) => {
+		use $benches::{wasm_binary_unwrap, Block};
+		pub fn main() {
+			let output = $crate::bench_runner::run::<Block>(wasm_binary_unwrap().to_vec());
+			$crate::handler::handle(output);
+		}
+	};
+}
+
+/// Define benches
+///
+/// Create a file `src/benches.rs`:
+/// ```.ignore
+/// #![cfg_attr(not(feature = "std"), no_std)]
+/// #![allow(dead_code)]
+///
+/// #[cfg(feature = "std")] // Re-export for bench_runner
+/// pub use crate::mock::{Block, wasm_binary_unwrap};
+///
+/// use crate::mock::YourModule;
+///
+/// fn foo(b: &mut Bencher) {
+///     b.bench("foo", || {
+///         YourModule::foo();
+///     });
+/// }
+///
+/// fn bar(b: &mut Bencher) {
+///     b.bench("bar", || {
+///         YourModule::bar();
+///     });
+/// }
+///
+/// orml_bencher::bench!(foo, bar);
+/// ```
+/// Update `src/lib.rs`:
+/// ```.ignore
+/// #[cfg(any(feature = "bench", test))]
+/// pub mod mock; /* mock runtime needs to be compiled into wasm */
+/// #[cfg(feature = "bench")]
+/// pub mod benches;
+/// ```
+#[macro_export]
+macro_rules! bench {
+    (
+        $($method:path),+
+    ) => {
+        use $crate::BenchResult;
+        use $crate::sp_std::{cmp::max, prelude::Vec};
+        use $crate::frame_benchmarking::{benchmarking, BenchmarkResults};
+
+        #[derive(Default, Clone, PartialEq, Debug)]
+        struct Bencher {
+            pub results: Vec<BenchResult>,
+        }
+
+        impl Bencher {
+            pub fn bench<F: Fn() -> ()>(&mut self, name: &str, block: F) {
+                // Warm up the DB
+                benchmarking::commit_db();
+                benchmarking::wipe_db();
+
+                let mut result = BenchResult {
+                    method: name.as_bytes().to_vec(),
+                    ..Default::default()
+                };
+
+                for _ in 0..50 {
+                    benchmarking::commit_db();
+                    benchmarking::reset_read_write_count();
+
+                    let start_time = benchmarking::current_time();
+                    block();
+                    let end_time = benchmarking::current_time();
+                    let elasped = end_time - start_time;
+                    result.elapses.push(elasped);
+
+                    benchmarking::commit_db();
+                    let (reads, repeat_reads, writes, repeat_writes) = benchmarking::read_write_count();
+
+                    result.reads = max(result.reads, reads);
+                    result.repeat_reads = max(result.repeat_reads, repeat_reads);
+                    result.writes = max(result.writes, writes);
+                    result.repeat_writes = max(result.repeat_writes, repeat_writes);
+
+                    benchmarking::wipe_db();
+                }
+                self.results.push(result);
+            }
+        }
+
+        $crate::sp_core::wasm_export_functions! {
+            fn run_benches() -> Vec<BenchResult> {
+                let mut bencher = Bencher::default();
+                $(
+                    $method(&mut bencher);
+                )+
+                bencher.results
+            }
+        }
+    }
+}
diff --git a/weight-gen/Cargo.toml b/weight-gen/Cargo.toml
@@ -0,0 +1,17 @@
+[package]
+name = "weight-gen"
+description = "CLI for generating weight from bencher output"
+license = "Apache-2.0"
+version = "0.4.1-dev"
+authors = ["Laminar Developers <hello@laminar.one>"]
+edition = "2018"
+
+[dependencies]
+serde = { version = "1.0.119", features = ['derive'] }
+serde_json = "1.0"
+clap = "3.0.0-beta.2"
+handlebars = { version = "3.5.2" }
+
+[features]
+default = ["std"]
+std = []