diff --git a/doc/substrate/README.md b/doc/substrate/README.md new file mode 100644 index 00000000..51288d65 --- /dev/null +++ b/doc/substrate/README.md @@ -0,0 +1,99 @@ +# Scout Substrate: Security Analysis Tool + +Scout is an extensible open-source tool intended to assist Substrate developers and auditors detect common security issues and deviations from best practices. + +This tool will help developers write secure and more robust Substrate pallets, runetime code and node code. + +Our interest in this project comes from our experience in manual auditing, our usage of comparable tools, and the development of Scout for smart contracts in [Polkadot ink!](https://github.com/CoinFabrik/scout) and [other blockchains](https://github.com/CoinFabrik/scout). + +## Quick Start + +**Install Scout Audit** + +Make sure that [Cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html) is installed on your computer. Then, install Scout with the following command: + +```console +cargo install cargo-scout-audit +``` + +**Run Scout Audit:** + +To run Scout on your project execute the following command: + +```console +cargo scout-audit +``` + +## Detectors + +Currently, Scout for Substrate includes the following detectors. + +| Detector | What it detects | Test cases (vulnerable/remediated) | +| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| avoid-dispatch-error | Usage of `DispatchError::Other` for error codes. | [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/avoid-dispatcherror-other/vulnerable/vulnerable-1) / [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/avoid-dispatcherror-other/remediated/remediated-1) | +| integer-overflow-or-underflow | Potential for integer arithmetic overflow/underflow. | [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/integer-overflow-or-underflow/vulnerable/vulnerable-1), [2](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/integer-overflow-or-underflow/vulnerable/vulnerable-2) / [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/integer-overflow-or-underflow/remediated/remediated-1), [2](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/integer-overflow-or-underflow/remediated/remediated-2) | +| unsafe-expect | Unsafe usage of `expect`. | [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/unsafe-expect/vulnerable/vulnerable-1) / [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/unsafe-expect/remediated) | +| unsafe-unwrap | Unsafe usage of `unwrap`. | [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/unsafe-unwrap/vulnerable/vulnerable-1) / [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/unsafe-unwrap/remediated) | +| known-vulnerabilities | Usage of dependencies with known vulnerabilities. | [1](./test-cases/known-vulnerabilities/vulnerable/vulnerable-1/) / [1](./test-cases/known-vulnerabilities/remediated/remediated-1/) | +| avoid-panic-error | Code panics on error instead of using descriptive enum. | [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/avoid-panic-error/vulnerable/vulnerable-1) / [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/avoid-panic-error/remediated/remediated-1) | +| iterators-over-indexing | Iterating with hardcoded indexes is slower than using an iterator. Also, if the index is out of bounds, it will panic. | [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/iterators-over-indexing/vulnerable/vulnerable-1) / [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/iterators-over-indexing/remediated/remediated-1) | +| overflow-check | An arithmetic operation overflows or underflows the available memory allocated to the variable. | +| divide-before-multiply | Performing a division operation before a multiplication, leading to loss of precision. | [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/divide-before-multiply/vulnerable/vulnerable-1) / [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/divide-before-multiply/remediated/remediated-1) | +| incorrect-exponentiation | Incorrect usage of `^`. | [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/incorrect-exponentiation/vulnerable/vulnerable-1) / [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/incorrect-exponentiation/remediated/remediated-1) | +| avoid-unsafe-block | Using unsafe blocks in risks code safety and reliability. | [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/avoid-unsafe-block/vulnerable/vulnerable-1) / [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/avoid-unsafe-block/remediated/remediated-1) | +| assert-violation | Usage of the macro `assert!`, which can panic. | [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/assert-violation/vulnerable/vulnerable-1) / [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/assert-violation/remediated/remediated-1) | +| empty-expect | Use of `expect` with an empty string. | [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/empty-expect/vulnerable/vulnerable-1) / [1](https://github.com/CoinFabrik/scout-audit/tree/main/test-cases/substrate-pallets/empty-expect/remediated/remediated-1) | + +## Data Set + +With the intention of sharing the results of our research, we have created a public repository with a collection of thoroughly analyzed audited Substrate pallets, runtime, and node code. This repository serves as a knowledge base for Substrate developers, auditors, and security researchers aiming to identify common Substrate vulnerabilities and improve the security of their projects. + +In addition, we've made the data set publicly available in Hugging Face. + +:point_right: [Data set repository](https://github.com/CoinFabrik/scout-substrate-dataset) + +:point_right: [Hugging Face data set](https://huggingface.co/datasets/CoinFabrik/scout-substrate-m1) + +:heavy_exclamation_mark: Please note that this is a work in progress. The dataset will be updated as we continue with security audit analyses, identify issues, and refine their categories. + +## Roadmap + +**Milestone 1 - Initial Vulnerability Data Set and PoC Detectors** :white_check_mark: + +- Data Set. An open-source GitHub repository containing annotated Substrate pallets, runtime code and node code, accompanied by a detailed document listing vulnerability classes and their locations within the code. The dataset will also be uploaded to Hugging Face data sets. +- Proof-of-concept version of the detectors. Building on our experience with Scout for ink! and Soroban, we aim to detect 4 vulnerability classes. The tool will be delivered as source code in the repository. + +**Milestone 2: Extended Vulnerability Data Set and Prototype Detectors with Precision and Recall** :white_check_mark: + +- Prototype version of the detectors. Given our experience developing Scout for ink! and Soroban we aim to include detection of at least 5 new issue classes. Delivered as source code in the repository. +- Extended Data Set. Extended repository of vulnerable pallets, also extending list of vulnerabilities. Updated Hugging Face data set. +- Open Source Framework, for running analysis tools on Rust code, publicly available on CoinFabrik’s GitHub repository. +- Detector’s evaluation report on benchmark Data Set. List of suggested vulnerability classes that appear as false negatives in the report, or have a high rate of false positives. (See an example of the evaluation report for Scout for Soroban) + +**Milestone 3: Prototype Tool Integration with CLI, VSCode, and CI/CD, Documentation** + +- A prototype tool that integrates built detectors with a CLI, a VSCode extension, and a CI/CD GitHub Action. (See existing VSCode extension and GitHub Action for ink! and Soroban) + Additional or improved detectors for problematic issues identified in Milestone 2. Given our experience developing and improving Scout for ink! and Soroban, we aim to improve or further develop 3 detectors. +- Comprehensive integration tests for all detectors and features. +- A Documentation Site (using Docusaurus or GitBook) detailing tool usage and an initial set of detectors, including nine documented detectors developed in Milestones 1 and 2. (See the documentation pages for Scout on ink! and Soroban) +- A public project GitHub repository and website, along with an alpha tool release for selected projects and users. + +**Milestone 4: Final Precision and Recall Evaluation & Full Tool Release** + +- Final precision and recall evaluation report. Responsible disclosure of any sensible findings to their corresponding projects. + Improved detectors based on evaluation results. Given our experience developing Scout for ink! and Soroban, we aim to improve or develop 2 detectors after this final precision and recall. +- Fully integrated tool with CLI, VSCode Extension, and/or CI/CD GitHub Action. +- Public release of the tool with full documentation, publicly available on documentation sites (Docusaurus or GitBook)(See documentation examples here and here). +- Video tutorials on how to use the tool, along with one video tutorial for each issue detected by the tool. Given our experience developing Scout for ink! and Soroban, we aim to publish between 10 and 15 video tutorials on CoinFabrik’s YouTube channel. (See Scout video tutorials for other blockchain here). +- Release Webinar. +- Posts on CoinFabrik’s social media. + +## About CoinFabrik + +We - [CoinFabrik](https://www.coinfabrik.com/) - are a research and development company specialized in Web3, with a strong background in cybersecurity. Founded in 2014, we have worked on over 500 blockchain-related projects, EVM based and also for Solana, Algorand, and Polkadot. Beyond development, we offer security audits through a dedicated in-house team of senior cybersecurity professionals, currently working on code in Substrate, Solidity, Clarity, Rust, and TEAL. + +Our team has an academic background in computer science and mathematics, with work experience focused on cybersecurity and software development, including academic publications, patents turned into products, and conference presentations. Furthermore, we have an ongoing collaboration on knowledge transfer and open-source projects with the University of Buenos Aires. + +## License + +Scout is licensed and distributed under a MIT license. [Contact us](https://www.coinfabrik.com/) if you're looking for an exception to the terms. diff --git a/doc/substrate/precision-and-recall.md b/doc/substrate/precision-and-recall.md new file mode 100644 index 00000000..3792d268 --- /dev/null +++ b/doc/substrate/precision-and-recall.md @@ -0,0 +1,65 @@ +# Scout on Substrate: Precision and Recall Report + +## 1. Introduction + +As part of the second milestone of our initiative to develop Scout static analyzer for Substrate, we aimed to assess Scout’s precision and recall across a diverse set of repositories. This analysis involved running Scout over multiple projects, including pallets and runtimes, and comparing its findings against audited and unaudited code segments to identify true positives, false positives, and compilation issues affecting the tool’s reliability. + +To keep track of these analyses and their details, we created a dedicated [Excel spreadsheet](https://docs.google.com/spreadsheets/d/1a7yfwU206FeSToDt74d-l6gSzhO9IYih6LK6e-07Ajo) containing the auditor’s notes, true positive/false positive classifications, and additional context on each finding. Request access if you are interested in following the analysis more closely. The selected issues from audited projects considered for this analysis were filtered from the [Scout Substrate Dataset](https://github.com/CoinFabrik/scout-substrate-dataset), ensuring that the exact location of each issue in the code was informed in the audit report. + +The version of Scout considered in this analysis corresponds to commit [54c2328caa0b65b30f31fc2e244ac2419a52f98f](https://github.com/CoinFabrik/scout-audit/tree/54c2328caa0b65b30f31fc2e244ac2419a52f98f). + +## 2. Compilation Errors + +During the evaluation process, Scout encountered several compilation issues when analyzing certain projects. Some minor issues were addressed by simply running `cargo update` to update dependencies. However, more complex errors were observed due to environmental incompatibilities, missing WASM targets, and misreporting statuses when the tool encountered compilation failures. + +### 2.1. Common Types of Compilation Errors + +| Type of Error | Description | Example/Note | Resolution Steps | +| ------------------------------------------- | ----------------------------------------------------------------------------------- | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Dependency Conflicts / Missing Dependencies | Crates failing to compile due to out-of-date or conflicting dependencies. | `proc-macro2` compilation error in Manta project | Update dependencies (e.g., `cargo update`) or adjust Cargo.toml | +| WASM-Related Issues / Missing WASM STD | Certain projects requiring WASM targets not properly configured in the environment. | Manta Network’s `wasm-test` environment issue | Install WASM targets, ensure proper `rustup` setup, update env. | +| Scout Misreporting Errors as “Analyzed” | Scout not accurately reflecting compilation failures in the summary report. | Manta Network crate showing "Analyzed" despite fail | Addressed in [GitHub Issue #202](https://github.com/CoinFabrik/scout-audit/issues/202). Fixed status reporting in [GitHub PR #208](https://github.com/CoinFabrik/scout-audit/pull/208) | +| Unresolved Rust Environment Configurations | Complex runtime environment or large monorepos causing partial build failures. | Parallel, Astar, ReefChain builds failing | Adjust tool environment, ensure all required toolchains present | + +### 2.2. Steps Taken to Address Issues + +Initial attempts to resolve dependency-related errors involved running `cargo update` and verifying the presence of required toolchains (e.g., installing `wasm32-unknown-unknown` targets). For more complex scenarios, like those found in the Manta Network repository, a dedicated GitHub issue was opened and the issue was resolved (See [GitHub Issue #202](https://github.com/CoinFabrik/scout-audit/issues/202) and [PR #208](https://github.com/CoinFabrik/scout-audit/pull/208)). This issue focused on ensuring Scout accurately reports the "Compilation errors" status when a crate fails to compile, rather than incorrectly labeling the crate as "Analyzed." + +In addition to resolving reporting inaccuracies, the development team examined environmental and configuration issues. They worked on updating Scout’s handling of non-standard build environments and improving the logic that determines when a project is considered successfully analyzed versus when it is blocked by compilation hurdles. + +These actions have enhanced Scout’s reliability and usability, ensuring that both minor and more systemic compilation issues are clearly surfaced and properly addressed in future analyses. + +## 3. Precision and Recall by Detector + +### 3.1. On Findings from Audited Projects + +Of the total **N = 47** issues analyzed from audited projects extracted from the Scout Substrate Dataset, **M = 35** corresponded to code segments that could not be fully analyzed due to compilation errors. As mentioned in section `2. Compilation Errors` above, these errors were reviewed and are being followed (See [GitHub Issue #202](https://github.com/CoinFabrik/scout-audit/issues/202) and [PR #208](https://github.com/CoinFabrik/scout-audit/pull/208)) to improve Scout. This left **N - M = 12** issues associated with code segments where Scout could run without errors. All 12 issues were associated with the `integer_overflow_or_underflow` detector. + +For these 12 issues, our auditor first reviewed each audit finding from which the analyzed code was extracted, determining whether Scout should detect the issue at the indicated line of code or not. + +Our auditor’s analysis revealed that 6 of these issues were true positives—Scout accurately detected a genuine issue—while the other 6 were true negatives, as the flagged issues, despite being highlighted in audits, did not represent a detection point for the actual vulnerability and where correctly not detected by Scout. + +### 3.2. On Scout Findings from Unaudited Projects + +In addition to analyzing issues extracted from audited projects, our auditor also reviewed Scout’s detections on newer commits from the same set of projects where the tool now compiled and ran successfully, but whose code segments were not previously audited. This resulted in the analysis of an additional set of **35** detections. + +For each detection, the auditor assessed whether it represented a true vulnerability (true positive) or if Scout incorrectly flagged a safe code pattern (false positive). In cases where the code was deemed non-vulnerable, and Scout should not have reported it, these detections were classified as false positives. No false negatives were considered here, as all findings listed are those that Scout flagged. + +The following table summarizes the results by detector: + +| Detector | True Positives | False Positives | % False Positives | +| ----------------------------- | -------------- | --------------- | ----------------- | +| known_vulnerabilities | 1 | 0 | 0% | +| avoid_panic_error | 2 | 0 | 0% | +| avoid-dispatcherror-other | 3 | 0 | 0% | +| assert_violation | 7 | 0 | 0% | +| unsafe_expect | 0 | 17 | 100% | +| integer_overflow_or_underflow | 0 | 5 | 100% | + +As shown above, while several detectors produced valid results with no false positives, others—particularly `unsafe_expect` and `integer_overflow_or_underflow`—yielded a high number of false positives. This insight helps guide future improvements to the detectors, aiming to reduce incorrect alerts and improve the overall reliability of Scout. + +## 4. Conclusion + +This precision and recall exercise is a critical component in refining Scout’s capabilities as we progress through Milestone 2 of our project. By running the tool against a variety of audited and unaudited projects, we have identified both the technical barriers—such as compilation errors and environmental setup issues—and the areas where our detectors need further refinement to minimize false positives and misclassifications. + +The results obtained here will inform the development of new detectors, improvements to existing ones, and enhancements to the supporting frameworks for analysis. Addressing these early findings will significantly improve Scout’s usability, ensuring that developers and auditors can rely on its insights. Ultimately, by identifying precision and environment-related issues now, we are setting a strong foundation for the subsequent milestones, which will deliver a more robust and accurate tool.