Skip to content
This repository has been archived by the owner on Nov 29, 2022. It is now read-only.

Import pared-down version of parity-util-mem into mysten-infra #134

Merged
merged 1 commit into from
Sep 4, 2022
Merged

Import pared-down version of parity-util-mem into mysten-infra #134

merged 1 commit into from
Sep 4, 2022

Conversation

aschran
Copy link
Contributor

@aschran aschran commented Aug 30, 2022

  • Removed all custom allocator features, leaving only estimate-heapsize.
  • Added MallocSizeOf implementaitons for some types used in narwhal.

@aschran aschran requested review from huitseeker and lanvidr August 30, 2022 20:14
@aschran aschran force-pushed the aschran-parity-util-mem branch 2 times, most recently from 6aa1d31 to 3801ab1 Compare August 30, 2022 21:19
@aschran aschran marked this pull request as ready for review August 30, 2022 22:50
@aschran
Copy link
Contributor Author

aschran commented Aug 31, 2022

There are a lot of lint/clippy errors. I don't think we should actually fix those errors because it will be easier to merge upstream commits if I leave the unchanged code as-is. Do you agree, or do you think it will be too hard/annoying for us to selectively disable the CI checks on this code?

@bmwill
Copy link
Contributor

bmwill commented Sep 1, 2022

It looks like you're forking an existing project. Would it make more sense to have that forked into a standalone repo so that we can better sync from upstream?

@aschran
Copy link
Contributor Author

aschran commented Sep 1, 2022

It looks like you're forking an existing project. Would it make more sense to have that forked into a standalone repo so that we can better sync from upstream?

I don't have a strong opinion on this. François suggested just copy-pasting it in here and then deleting all of the allocator features we don't want. I'm happy to follow whatever is the normal convention for something like this? Forking into a standalone repo might be nicer in that we wouldn't have all the issues with the CI config in this repo that doesn't necessarily apply to this imported code.

@huitseeker what do you think?

huitseeker
huitseeker reviewed Sep 1, 2022
View reviewed changes
Copy link
Contributor

@huitseeker huitseeker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. The justifications to fork here rather than another repo is we want fast trait implementations for our structures that may not make sense upstream (the IndexMap is one posterchild), and that will be subject to the orphan rule.
  2. I think the changes (1x/3 months) are not going to be that hard to integrate.
  3. I think we'll need to run some tooling on this code. Clippy may be avoided (I've left indications on how to deactivate inline), but we'll need at least a rustfmt run.
  4. We need a README at the root to explain why we need this, and how we intend to use it, and ideally a simple code example in a sibling directory of src. I know we have draft PRs using this downstream, but I'm thinking of catering to folks that don't have this context.
  5. Otherwise, this LGTM!

crates/parity-util-mem/derive/CHANGELOG.md Outdated
@@ -0,0 +1,8 @@
# Changelog
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I just kept it because I made zero modifications to the derive crate. But I can still scrap it if we are closing the door on trying to keep upstream merging easier (which I think we are, if we run rustfmt, etc, and your reasoning for this makes sense to me)

crates/parity-util-mem/derive/lib.rs Outdated Show resolved Hide resolved
crates/parity-util-mem/src/lib.rs Show resolved Hide resolved
@aschran aschran force-pushed the aschran-parity-util-mem branch from 3801ab1 to f073058 Compare September 1, 2022 17:59
@aschran
Copy link
Contributor Author

aschran commented Sep 1, 2022

  1. We need a README at the root to explain why we need this, and how we intend to use it, and ideally a simple code example in a sibling directory of src. I know we have draft PRs using this downstream, but I'm thinking of catering to folks that don't have this context.

My plan was to finish this and the PoC PR in narwhal, then come back and add a README guide at that point when all the details have been sorted out by virtue of having completed the "practice" implementation. Does that sound ok to you? I can create a first README draft immediately if you think this would be bad to initially merge without one.

@aschran aschran force-pushed the aschran-parity-util-mem branch from f073058 to bf74aef Compare September 1, 2022 18:03
@aschran
Copy link
Contributor Author

aschran commented Sep 1, 2022

@huitseeker is there a way to skip the crates-in-crates-directory check or should I restructure the layout of the imported code to pass it?

@bmwill
Copy link
Contributor

bmwill commented Sep 1, 2022 •
edited
Loading

@huitseeker is there a way to skip the crates-in-crates-directory check or should I restructure the layout of the imported code to pass it?

that check doesn't have a way to disable it for a particular case (which is intentional :) )

lanvidr
lanvidr approved these changes Sep 2, 2022
View reviewed changes
Copy link
Contributor

@lanvidr lanvidr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be more simple to keep this formatted as the rest of our code, and not worry about merging back upstream. Pending the += fix, lgtm!

crates/parity-util-mem/src/external_impls.rs Outdated
if let (Some(k), Some(v)) = (K::constant_size(), V::constant_size()) {
n += self.len() * (k + v)
} else {
n = self
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
n = self
n += self

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for catching this!

@aschran aschran force-pushed the aschran-parity-util-mem branch 2 times, most recently from 2b53727 to 07e78af Compare September 2, 2022 15:27
@huitseeker
Copy link
Contributor

But I can still scrap it if we are closing the door on trying to keep upstream merging easier

I think we are OK with scrapping it. The orphan rule will force us to put every implementation of the trait for a struct that's not accessible to us in this crate. "Easy upstream merges" is near-impossible anyway.

is there a way to skip the crates-in-crates-directory check or should I restructure the layout of the imported code to pass it?
We should rename the directory, I think.

@aschran aschran enabled auto-merge (squash) September 2, 2022 15:55
@aschran aschran force-pushed the aschran-parity-util-mem branch from 07e78af to f019492 Compare September 2, 2022 16:23
oxade
oxade reviewed Sep 2, 2022
View reviewed changes
crates/parity-util-mem/src/allocators.rs

} else if #[cfg(any(
target_os = "linux",
target_os = "android",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Surely we don't need this? :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah, I tried to remove all that crap that I could find but thanks for pointing this out.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done although it doesn't seem to be related to the windows rocksdb build failure.

(I made estimate-heapsize enabled by default so I think that should never actually be selected in any of our builds)

@aschran aschran force-pushed the aschran-parity-util-mem branch from f019492 to 2963277 Compare September 2, 2022 19:55
@aschran
Import pared-down version of parity-util-mem into mysten-infra
31034ca 
* Removed all custom allocator features, leaving only estimate-heapsize.
* Added MallocSizeOf implementaitons for some types used in narwhal.
@aschran aschran force-pushed the aschran-parity-util-mem branch from 2963277 to 31034ca Compare September 2, 2022 20:26
@aschran aschran merged commit cec9b78 into main Sep 4, 2022
@aschran aschran deleted the aschran-parity-util-mem branch September 4, 2022 04:50
sadhansood pushed a commit to MystenLabs/sui that referenced this pull request Nov 7, 2022
@aschran
Import pared-down version of parity-util-mem into mysten-infra (Myste…
62ef2fa 
…nLabs/mysten-infra#134)

* Removed all custom allocator features, leaving only estimate-heapsize.
* Added MallocSizeOf implementaitons for some types used in narwhal.
sadhansood pushed a commit to MystenLabs/sui that referenced this pull request Nov 7, 2022
@aschran @sadhansood
Import pared-down version of parity-util-mem into mysten-infra (Myste…
64ad33c 
…nLabs/mysten-infra#134)

* Removed all custom allocator features, leaving only estimate-heapsize.
* Added MallocSizeOf implementaitons for some types used in narwhal.
sadhansood pushed a commit to MystenLabs/sui that referenced this pull request Nov 9, 2022
@aschran @sadhansood
Import pared-down version of parity-util-mem into mysten-infra (Myste…
4e42f3c 
…nLabs/mysten-infra#134)

* Removed all custom allocator features, leaving only estimate-heapsize.
* Added MallocSizeOf implementaitons for some types used in narwhal.
sadhansood pushed a commit to MystenLabs/sui that referenced this pull request Nov 9, 2022
@aschran @sadhansood
Import pared-down version of parity-util-mem into mysten-infra (Myste…
0879eca 
…nLabs/mysten-infra#134)

* Removed all custom allocator features, leaving only estimate-heapsize.
* Added MallocSizeOf implementaitons for some types used in narwhal.
sadhansood added a commit to MystenLabs/sui that referenced this pull request Nov 9, 2022
@sadhansood @bmwill
@huitseeker @mystenmark @akichidis @velvia @lanvidr @oxade @punwai @mwtian @dependabot @longbowlu @aschran @andll @macalinao @erwanor @benr-ml @asonnino @brad-mysten @joyqvq @sblackshear
Sadhan/merge_mysten_infra (#5955)
01d6d31 
* tracing: introduce a logger for testing environments (MystenLabs/mysten-infra#76)

* chore: add docs, license, codecov badges

* ci: move cargo-udeps to nightly

* ci: use nextest to run rust tests

* tracing: improve default and testing subscribers (MystenLabs/mysten-infra#79)

Improve the default and testing subscribers to include more information
like the line number as well as outputting span creation and close
events.

* chore: update CODEOWNERS

* tracing: send logging to stderr instead of stdout by default

In many cases, e.g. cli tools like the wallet, a tool may want to print
out some useful information to stdout. Today we currently log to stdout
which makes it very difficult to distinguish between output we want the
user to actually use and log messages. In order to address this this
patch changes our logging infrastructure to log to stderr by default
instead of stdout.

* tracing: enable configuration of tracing via env vars

* network: fix build on windows

* ci: run tests on windows

* network: fix warnings on windows

* Remove colors from log output when stdout is not tty; remove useless log messages (MystenLabs/mysten-infra#85)

* Remove unhelpful log messages

* Don't use ansi colors unless stderr is a tty

* [mysten_network] add support for metrics reporting (MystenLabs/mysten-infra#83)

This commit is introducing an interface to act as a callback for performed requests against the server. Then those callbacks can be used for metrics reporting purposes.

* port to rust 1.62

* refactor: convert exact instances of then_some

* chore: clippy run

* chore: one round of updates

* deactivate all-features in doctests

* fix: force http version & remove Lazy

* Use multi-threaded-cf feature so we can get perf stats (MystenLabs/mysten-infra#89)

* add iter all function to typed-store (MystenLabs/mysten-infra#91)

* Add opening as secondary (MystenLabs/mysten-infra#90)

* Add opening as secondary

* feat: add an optional predicate to table iteration

As this is liable to overload memory, the present PR introduces an optional Predicate argument (a `Fn(&(Key, Value)) -> bool`) which allows filtering a subset of the table's items instead of the whole table.

* tracing: wrap EnvFilter layer in reloadable layer (MystenLabs/mysten-infra#94)

Also fix our misuse of layering various optional tracing layers.

* chore: allow Unicode-DFS-2016 license

* chore: one round of updates

* feat: make the log string overridable in TelemetrySubscriber

* tracing: improve the panic handler (MystenLabs/mysten-infra#96)

Improve our panic handler to do the following:

Also call the stdlib's panic handler so backtraces aren't swallowed
flush stdout and stderr
enable configuring if the panic handler should crash the process

* [crypto] Create verifier of X.509 `SubjectPublicKeyInfo` against a collection of known PKs. (MystenLabs/mysten-infra#100)

* add PskSet

* Add a ultility function to convert `MultiAddr` to `SocketAddr` (MystenLabs/mysten-infra#101)

* add a ultility function to convert MultiAddr to SocketAddr

* fixup! add a ultility function to convert MultiAddr to SocketAddr

* add negative test

* fixup! add negative test

* DBMap utils auto derive: reduce boilerplate functions (MystenLabs/mysten-infra#103)

* DBMap utils

* feat(ci): auto-merge certain dependabot PRs

* chore(deps): one round of updates

* Support generics (MystenLabs/mysten-infra#108)

* support generics

* Set default total memtable size to 512MB per Rocksdb (MystenLabs/mysten-infra#109)

* chore(ci): Auto-mark (and then close) stale issues / PRs (MystenLabs/mysten-infra#111)

Issues/PRs that are:
    - over 60 days w/o activity,
    - not assigned,
    - not in a milestone

    Get automatically closed after a 7 days grace period.

* chore(deps): update rocksdb requirement from 0.18.0 to 0.19.0 (MystenLabs/mysten-infra#113)

* fix: remove the anyhow backtrace feature

the workspace-hack from cargo-hakari unifies the versions we use to a major version number (1) =>
we hit dtolnay/anyhow#250

This removes the `backtrace` feature.

* Set default max WAL size per RocksDB and allow configuration via env var (MystenLabs/mysten-infra#118)

* set max wal size. allow configuration via env var

* fixup! set max wal size. allow configuration via env var

* chore(deps): update crossterm requirement from 0.24.0 to 0.25.0 (MystenLabs/mysten-infra#122)

* chore: bump rust toolchain to 1.63 (MystenLabs/mysten-infra#121)

* Cleanup: use one expression to read usize from env var (MystenLabs/mysten-infra#119)

* refactor: replace anyhow by eyre everywhere (MystenLabs/mysten-infra#117)

Better architecture of the import of backtrace-rs, which allows its use on stable.
Plenty of other features (colors in logs!), and unification of the libs we use across the architecture.

* Allow specifying each tables options, and getting mem usage (MystenLabs/mysten-infra#116)

* add configurator, user specified opts, mem stats

* refactor: fold the typed_store_macros crate inside typed_store (MystenLabs/mysten-infra#127)

The organization fits the classical crate organization idiom better.

* [typed-store] implement the reverse() on iterator (MystenLabs/mysten-infra#125)

Introduce the RevIter so we can traverse the iteration naturally in reverse order

* debug print when rocksdb option config not found in env rather than info print (MystenLabs/mysten-infra#130)

* utils for Store open and config (MystenLabs/mysten-infra#123)

* utils for Store open and config
* organize shared logic better

* [Feature][Telemetry] Automatic span latency measurement (MystenLabs/mysten-infra#131)

* add fn to get raw bytes (MystenLabs/mysten-infra#137)

* rename get_or_insert to indicate unsafe (MystenLabs/mysten-infra#138)

* Import pared-down version of parity-util-mem into mysten-infra (MystenLabs/mysten-infra#134)

* Removed all custom allocator features, leaving only estimate-heapsize.
* Added MallocSizeOf implementaitons for some types used in narwhal.

* Add `MallocSizeOf` impl for `RoaringBitmap` (MystenLabs/mysten-infra#140)

* Add `MallocSizeOf` impl for `Ed25519AggregateSignature` (MystenLabs/mysten-infra#141)

* Allow to track request lifecycle via MetricsCallbackProvider (MystenLabs/mysten-infra#142)

This provides two simple methods to introspect start and end of the request.

* refactor: rename typed-store-macros -> typed-store-derive

* refactor: rename parity-util-mem -> mysten-util-mem

* Update roaring in mysten-mem-util to 0.10.0 (MystenLabs/mysten-infra#144)

* feat: Publishing tools for mysten-infra crates (MystenLabs/mysten-infra#145)

* chore: fix typos job

* feat: add publishing tooling

* fix: make typed-store publishable

* chore: make name-variant publishable

* chore: make telemetry-subscribers publishable

* chore: make mysten-network publishable

* feat(ci): autopublish scripts

* fixup! feat(ci): autopublish scripts

* fixup! feat: add publishing tooling

* fixup! feat: add publishing tooling

* Update `fastcrypto` in `mysten-util-mem` to 0.1 (MystenLabs/mysten-infra#146)

* Update `fastcrypto` in `mysten-util-mem` to 0.1.1 (MystenLabs/mysten-infra#147)

* Update to Tonic v0.8.0 (MystenLabs/mysten-infra#139)

* Maintenance (repair nightly) (MystenLabs/mysten-infra#151)

* chore(deps): remove unneeded dependencies

* fix typos job

* fixup! chore(deps): remove unneeded dependencies

* chore(deps): update lru requirement from 0.7 to 0.8 (MystenLabs/mysten-infra#152)

Updates the requirements on [lru](https://github.com/jeromefroe/lru-rs) to permit the latest version.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases)
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md)
- [Commits](jeromefroe/lru-rs@0.7.0...0.8.0)

---
updated-dependencies:
- dependency-name: lru
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add malloc sizes for BLS crypto types. (MystenLabs/mysten-infra#153)

* Add `README.md` for `mysten-util-mem` (MystenLabs/mysten-infra#155)

* chore(deps): bump codecov/codecov-action from 1 to 3 (MystenLabs/mysten-infra#159)

Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 1 to 3.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](codecov/codecov-action@v1...v3)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump actions/checkout from 2 to 3 (MystenLabs/mysten-infra#158)

Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [feature] allow verification against externally signed certificates

* small fixes to publish scripts

* Add a `ClosureMetric` for Prometheus (MystenLabs/mysten-infra#160)

Implements a `ClosureMetric` for crate `prometheus` whose value is computed at
the time of collection by a provided closure.

* chore(deps): bump actions/stale from 5 to 6 (MystenLabs/mysten-infra#165)

Bumps [actions/stale](https://github.com/actions/stale) from 5 to 6.
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](actions/stale@v5...v6)

---
updated-dependencies:
- dependency-name: actions/stale
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat(typed-store): default implementations for `Map::multi_*` (MystenLabs/mysten-infra#161)

* feat(typed-store): default implementations for `Map::multi_*`

* feat(typed-store): document that default impls are non-atomic

* chore(deps): one round of updates (MystenLabs/mysten-infra#166)

* chore(deps): update fastcrypto requirement from 0.1.1 to 1.0.0 (MystenLabs/mysten-infra#162)

Updates the requirements on [fastcrypto](https://github.com/MystenLabs/fastcrypto) to permit the latest version.
- [Release notes](https://github.com/MystenLabs/fastcrypto/releases)
- [Commits](MystenLabs/fastcrypto@fastcrypto-v0.1.1...fastcrypto-v1.0.0)

---
updated-dependencies:
- dependency-name: fastcrypto
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [Telemetry-subscribers] Per-layer/separate log and span filtering; fix Tokio Console (MystenLabs/mysten-infra#167)

* fix(ci): make nightly availablein publish (MystenLabs/mysten-infra#168)

* chore(deps): update rcgen requirement from 0.9.3 to 0.10.0 (MystenLabs/mysten-infra#170)

* chore(deps): update clap requirement from 3.1.14 to 4.0.8 (MystenLabs/mysten-infra#169)

Updates the requirements on [clap](https://github.com/clap-rs/clap) to permit the latest version.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](clap-rs/clap@clap_complete-v3.2.0...v4.0.8)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: publish mysten-network post MystenLabs/mysten-infra#139 (MystenLabs/mysten-infra#173)

* Change the dependency on fastcrypto to a specific commit (MystenLabs/mysten-infra#175)

Change the dependency on fastcrypto to a specific commit

* Add user metrics for tracking db operations (MystenLabs/mysten-infra#172)

* Publish new version of typed store and typed store derive (MystenLabs/mysten-infra#176)

* Blocking write (MystenLabs/mysten-infra#174)

Blocking write

* chore(deps): upgrade minor versions, fix unused env key (MystenLabs/mysten-infra#178)

* chore: update nexlint (MystenLabs/mysten-infra#183)

This updates nexlint in order to pick up a dependency update in guppy, in order to fix errors that are cropping up in ci:

```
Error: while building package graph

Caused by:
    failed to construct package graph: for package 'windows-sys 0.42.0 (registry+https://github.com/rust-lang/crates.io-index)': for dependency 'windows_aarch64_gnullvm', parsing target 'aarch64-pc-windows-gnullvm' failed: unknown target triple
```

* trigger new release (MystenLabs/mysten-infra#182)

* fix: stop auto-closing issues (MystenLabs/mysten-infra#184)

* chore(deps): update multiaddr requirement from 0.14.0 to 0.15.0 (MystenLabs/mysten-infra#186)

* Add sampling for metrics reporting (MystenLabs/mysten-infra#179)

* Sadhan/add_cf_metrics (MystenLabs/mysten-infra#180)

* Add metrics per column family in DBMap

* Add column family metrics

* Add cf metrics

* Add rocksdb perf context metrics (MystenLabs/mysten-infra#181)

* Update fastcrypto (MystenLabs/mysten-infra#191)

* Fix how we init db metrics (MystenLabs/mysten-infra#189)

* Make sync use full feature (MystenLabs/mysten-infra#192)

* ci: run on {windows,ubuntu}-ghcloud (MystenLabs/mysten-infra#190)

* Publish new typed store (MystenLabs/mysten-infra#188)

* Update typed-store version to v0.4.0 (MystenLabs/mysten-infra#193)

* chore: update fastcrypto (MystenLabs/mysten-infra#194)

* codec: use bincode::deserialize instead of deserialize_from (MystenLabs/mysten-infra#195)

* Update FastCrypto (MystenLabs/mysten-infra#196)

* feat: add MallocSizeOf for OnceCell (MystenLabs/mysten-infra#197)

* chore(deps): remove unused dep (MystenLabs/mysten-infra#199)

* chore: update fastcrypto (MystenLabs/mysten-infra#198)

* Merge mysten infra in sui

* Initial commit

* feat: Add DB interface crate

This adds a type-safe interface to access a rocksDB instance, with a HashMap-like DBMap trait,
where key-value pairings of a given type are stored under a specific columnfamily.

* chore: activate lto in release / bench

* chore: run unused dependency detection in CI

* chore: CI to use more reliable actions-rs/install

* fix: fix build of rust-rocksdb with a patch dependency (MystenLabs/mysten-infra#17)

rocksdb and librocksdb-sys depend on each other and since a recent upstream commit, rocksdb point at an outdated version of librocksdb-sys. This intercepts the versions they query out of crates.io and puts them in agreement.

* feat: add a certificate verifier matching an expecte pre-shared key

This allows accepting specifically correctly self-signed certificates which signer is
an expected signer.

* fix: clarify licenses for rccheck

* chore: remove obsolete workaround for rust-rocksdb/rust-rocksdb#602

* chore: disallow unbounded channels

* chore: fix license headers, add checking to CI

* chore: add dependabot configuration file

* feat: add crate that allows printing the name of an enum variant

* fix: add benign BSD-2-Clause to allowed licenses

* Laura/components (MystenLabs/mysten-infra#39)

* initial functions added

work in progress

use trait and implement join, terminate, restart

added the run function that listens for signals

* started on example user of the component library

* pair programming w/ francois

remove recursion

repair the select macros & terminate

* mocked the tcp stream so example is runable

updated use of cancellation channel

* make failure happen only once

* add doccumentation

* updates based on review

Co-authored-by: François Garillot <francois@garillot.net>

* fix: run clippy on tests as well

* New telemetry module: Shared logging/tracing initialization (MystenLabs/mysten-infra#45)

This creates a new module for common telemetry subscribers: Jaeger distributed tracing, logging including file output with daily rotation, Tokio console and more.

* Rename telemetry module to telemetry-subscribers (MystenLabs/mysten-infra#47)

* Rename project telemetry_subscribers

* Enable setting log level in config

* fmt and clippy

* chore: use a caching crate install

https://github.com/baptiste0928/cargo-install caches the binary after compilation, for speed of repeated execution

* chore: add code coverage

* ci: improve stability and caching

Make a number of improvements to ci similar to what were done here:
#1164

* rust: update to 1.60.0

* chore: clean up clippy.toml

* x: introduce nexlint support through

* lint: enforce crate names and paths

* lint: enforce that crates are in the  directory

* lint: enforce no trailing whitespace

* rccheck: pin ed25519 version due to semver incompatability with pkcs8 feature

* fix: remove the beta version in codecov job

* feat: generate & publish code doc

* fix: Bump ed25519 to 1.5.0, pkcs8 to 0.9

* fix: rename pedantic CI job

* chore: add a basic CODEOWNERS

* mysten-network: add common networking utilities

This adds the `mysten-network` crate which can be used to house common
networking tools across mysten repos. In particular this currently
includes:
- A tonic Codec for the bincode format.
- Utilites and configuration for building tonic servers or clients using
  the `multiaddr` format.

Supported multiaddr protocol stacks are:
- dns/tcp/{http,https}
- ip4/tcp/{http,https}
- ip6/tcp/{http,https}
- unix/{http,https}

* mysten-network: support cancelation of a server

* mysten-network: add some basic tests for using dns, ip4, ip6, and unix

* feat: instrument main RocksDB functions to get tracing output (MystenLabs/mysten-infra#74)

* ci: make dependabot update GH actions

* ci: fix typos in docs

* tracing: introduce a logger for testing environments (MystenLabs/mysten-infra#76)

* chore: add docs, license, codecov badges

* ci: move cargo-udeps to nightly

* ci: use nextest to run rust tests

* tracing: improve default and testing subscribers (MystenLabs/mysten-infra#79)

Improve the default and testing subscribers to include more information
like the line number as well as outputting span creation and close
events.

* chore: update CODEOWNERS

* tracing: send logging to stderr instead of stdout by default

In many cases, e.g. cli tools like the wallet, a tool may want to print
out some useful information to stdout. Today we currently log to stdout
which makes it very difficult to distinguish between output we want the
user to actually use and log messages. In order to address this this
patch changes our logging infrastructure to log to stderr by default
instead of stdout.

* tracing: enable configuration of tracing via env vars

* network: fix build on windows

* ci: run tests on windows

* network: fix warnings on windows

* Remove colors from log output when stdout is not tty; remove useless log messages (MystenLabs/mysten-infra#85)

* Remove unhelpful log messages

* Don't use ansi colors unless stderr is a tty

* [mysten_network] add support for metrics reporting (MystenLabs/mysten-infra#83)

This commit is introducing an interface to act as a callback for performed requests against the server. Then those callbacks can be used for metrics reporting purposes.

* port to rust 1.62

* refactor: convert exact instances of then_some

* chore: clippy run

* chore: one round of updates

* deactivate all-features in doctests

* fix: force http version & remove Lazy

* Use multi-threaded-cf feature so we can get perf stats (MystenLabs/mysten-infra#89)

* add iter all function to typed-store (MystenLabs/mysten-infra#91)

* Add opening as secondary (MystenLabs/mysten-infra#90)

* Add opening as secondary

* feat: add an optional predicate to table iteration

As this is liable to overload memory, the present PR introduces an optional Predicate argument (a `Fn(&(Key, Value)) -> bool`) which allows filtering a subset of the table's items instead of the whole table.

* tracing: wrap EnvFilter layer in reloadable layer (MystenLabs/mysten-infra#94)

Also fix our misuse of layering various optional tracing layers.

* chore: allow Unicode-DFS-2016 license

* chore: one round of updates

* feat: make the log string overridable in TelemetrySubscriber

* tracing: improve the panic handler (MystenLabs/mysten-infra#96)

Improve our panic handler to do the following:

Also call the stdlib's panic handler so backtraces aren't swallowed
flush stdout and stderr
enable configuring if the panic handler should crash the process

* [crypto] Create verifier of X.509 `SubjectPublicKeyInfo` against a collection of known PKs. (MystenLabs/mysten-infra#100)

* add PskSet

* Add a ultility function to convert `MultiAddr` to `SocketAddr` (MystenLabs/mysten-infra#101)

* add a ultility function to convert MultiAddr to SocketAddr

* fixup! add a ultility function to convert MultiAddr to SocketAddr

* add negative test

* fixup! add negative test

* DBMap utils auto derive: reduce boilerplate functions (MystenLabs/mysten-infra#103)

* DBMap utils

* feat(ci): auto-merge certain dependabot PRs

* chore(deps): one round of updates

* Support generics (MystenLabs/mysten-infra#108)

* support generics

* Set default total memtable size to 512MB per Rocksdb (MystenLabs/mysten-infra#109)

* chore(ci): Auto-mark (and then close) stale issues / PRs (MystenLabs/mysten-infra#111)

Issues/PRs that are:
    - over 60 days w/o activity,
    - not assigned,
    - not in a milestone

    Get automatically closed after a 7 days grace period.

* chore(deps): update rocksdb requirement from 0.18.0 to 0.19.0 (MystenLabs/mysten-infra#113)

* fix: remove the anyhow backtrace feature

the workspace-hack from cargo-hakari unifies the versions we use to a major version number (1) =>
we hit dtolnay/anyhow#250

This removes the `backtrace` feature.

* Set default max WAL size per RocksDB and allow configuration via env var (MystenLabs/mysten-infra#118)

* set max wal size. allow configuration via env var

* fixup! set max wal size. allow configuration via env var

* chore(deps): update crossterm requirement from 0.24.0 to 0.25.0 (MystenLabs/mysten-infra#122)

* chore: bump rust toolchain to 1.63 (MystenLabs/mysten-infra#121)

* Cleanup: use one expression to read usize from env var (MystenLabs/mysten-infra#119)

* refactor: replace anyhow by eyre everywhere (MystenLabs/mysten-infra#117)

Better architecture of the import of backtrace-rs, which allows its use on stable.
Plenty of other features (colors in logs!), and unification of the libs we use across the architecture.

* Allow specifying each tables options, and getting mem usage (MystenLabs/mysten-infra#116)

* add configurator, user specified opts, mem stats

* refactor: fold the typed_store_macros crate inside typed_store (MystenLabs/mysten-infra#127)

The organization fits the classical crate organization idiom better.

* [typed-store] implement the reverse() on iterator (MystenLabs/mysten-infra#125)

Introduce the RevIter so we can traverse the iteration naturally in reverse order

* debug print when rocksdb option config not found in env rather than info print (MystenLabs/mysten-infra#130)

* utils for Store open and config (MystenLabs/mysten-infra#123)

* utils for Store open and config
* organize shared logic better

* [Feature][Telemetry] Automatic span latency measurement (MystenLabs/mysten-infra#131)

* add fn to get raw bytes (MystenLabs/mysten-infra#137)

* rename get_or_insert to indicate unsafe (MystenLabs/mysten-infra#138)

* Import pared-down version of parity-util-mem into mysten-infra (MystenLabs/mysten-infra#134)

* Removed all custom allocator features, leaving only estimate-heapsize.
* Added MallocSizeOf implementaitons for some types used in narwhal.

* Add `MallocSizeOf` impl for `RoaringBitmap` (MystenLabs/mysten-infra#140)

* Add `MallocSizeOf` impl for `Ed25519AggregateSignature` (MystenLabs/mysten-infra#141)

* Allow to track request lifecycle via MetricsCallbackProvider (MystenLabs/mysten-infra#142)

This provides two simple methods to introspect start and end of the request.

* refactor: rename typed-store-macros -> typed-store-derive

* refactor: rename parity-util-mem -> mysten-util-mem

* Update roaring in mysten-mem-util to 0.10.0 (MystenLabs/mysten-infra#144)

* feat: Publishing tools for mysten-infra crates (MystenLabs/mysten-infra#145)

* chore: fix typos job

* feat: add publishing tooling

* fix: make typed-store publishable

* chore: make name-variant publishable

* chore: make telemetry-subscribers publishable

* chore: make mysten-network publishable

* feat(ci): autopublish scripts

* fixup! feat(ci): autopublish scripts

* fixup! feat: add publishing tooling

* fixup! feat: add publishing tooling

* Update `fastcrypto` in `mysten-util-mem` to 0.1 (MystenLabs/mysten-infra#146)

* Update `fastcrypto` in `mysten-util-mem` to 0.1.1 (MystenLabs/mysten-infra#147)

* Update to Tonic v0.8.0 (MystenLabs/mysten-infra#139)

* Maintenance (repair nightly) (MystenLabs/mysten-infra#151)

* chore(deps): remove unneeded dependencies

* fix typos job

* fixup! chore(deps): remove unneeded dependencies

* chore(deps): update lru requirement from 0.7 to 0.8 (MystenLabs/mysten-infra#152)

Updates the requirements on [lru](https://github.com/jeromefroe/lru-rs) to permit the latest version.
- [Release notes](https://github.com/jeromefroe/lru-rs/releases)
- [Changelog](https://github.com/jeromefroe/lru-rs/blob/master/CHANGELOG.md)
- [Commits](jeromefroe/lru-rs@0.7.0...0.8.0)

---
updated-dependencies:
- dependency-name: lru
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add malloc sizes for BLS crypto types. (MystenLabs/mysten-infra#153)

* Add `README.md` for `mysten-util-mem` (MystenLabs/mysten-infra#155)

* chore(deps): bump codecov/codecov-action from 1 to 3 (MystenLabs/mysten-infra#159)

Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 1 to 3.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](codecov/codecov-action@v1...v3)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump actions/checkout from 2 to 3 (MystenLabs/mysten-infra#158)

Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [feature] allow verification against externally signed certificates

* small fixes to publish scripts

* Add a `ClosureMetric` for Prometheus (MystenLabs/mysten-infra#160)

Implements a `ClosureMetric` for crate `prometheus` whose value is computed at
the time of collection by a provided closure.

* chore(deps): bump actions/stale from 5 to 6 (MystenLabs/mysten-infra#165)

Bumps [actions/stale](https://github.com/actions/stale) from 5 to 6.
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](actions/stale@v5...v6)

---
updated-dependencies:
- dependency-name: actions/stale
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat(typed-store): default implementations for `Map::multi_*` (MystenLabs/mysten-infra#161)

* feat(typed-store): default implementations for `Map::multi_*`

* feat(typed-store): document that default impls are non-atomic

* chore(deps): one round of updates (MystenLabs/mysten-infra#166)

* chore(deps): update fastcrypto requirement from 0.1.1 to 1.0.0 (MystenLabs/mysten-infra#162)

Updates the requirements on [fastcrypto](https://github.com/MystenLabs/fastcrypto) to permit the latest version.
- [Release notes](https://github.com/MystenLabs/fastcrypto/releases)
- [Commits](MystenLabs/fastcrypto@fastcrypto-v0.1.1...fastcrypto-v1.0.0)

---
updated-dependencies:
- dependency-name: fastcrypto
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [Telemetry-subscribers] Per-layer/separate log and span filtering; fix Tokio Console (MystenLabs/mysten-infra#167)

* fix(ci): make nightly availablein publish (MystenLabs/mysten-infra#168)

* chore(deps): update rcgen requirement from 0.9.3 to 0.10.0 (MystenLabs/mysten-infra#170)

* chore(deps): update clap requirement from 3.1.14 to 4.0.8 (MystenLabs/mysten-infra#169)

Updates the requirements on [clap](https://github.com/clap-rs/clap) to permit the latest version.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](clap-rs/clap@clap_complete-v3.2.0...v4.0.8)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: publish mysten-network post MystenLabs/mysten-infra#139 (MystenLabs/mysten-infra#173)

* Change the dependency on fastcrypto to a specific commit (MystenLabs/mysten-infra#175)

Change the dependency on fastcrypto to a specific commit

* Add user metrics for tracking db operations (MystenLabs/mysten-infra#172)

* Publish new version of typed store and typed store derive (MystenLabs/mysten-infra#176)

* Blocking write (MystenLabs/mysten-infra#174)

Blocking write

* chore(deps): upgrade minor versions, fix unused env key (MystenLabs/mysten-infra#178)

* chore: update nexlint (MystenLabs/mysten-infra#183)

This updates nexlint in order to pick up a dependency update in guppy, in order to fix errors that are cropping up in ci:

```
Error: while building package graph

Caused by:
    failed to construct package graph: for package 'windows-sys 0.42.0 (registry+https://github.com/rust-lang/crates.io-index)': for dependency 'windows_aarch64_gnullvm', parsing target 'aarch64-pc-windows-gnullvm' failed: unknown target triple
```

* trigger new release (MystenLabs/mysten-infra#182)

* fix: stop auto-closing issues (MystenLabs/mysten-infra#184)

* chore(deps): update multiaddr requirement from 0.14.0 to 0.15.0 (MystenLabs/mysten-infra#186)

* Add sampling for metrics reporting (MystenLabs/mysten-infra#179)

* Sadhan/add_cf_metrics (MystenLabs/mysten-infra#180)

* Add metrics per column family in DBMap

* Add column family metrics

* Add cf metrics

* Add rocksdb perf context metrics (MystenLabs/mysten-infra#181)

* Update fastcrypto (MystenLabs/mysten-infra#191)

* Fix how we init db metrics (MystenLabs/mysten-infra#189)

* Make sync use full feature (MystenLabs/mysten-infra#192)

* ci: run on {windows,ubuntu}-ghcloud (MystenLabs/mysten-infra#190)

* Publish new typed store (MystenLabs/mysten-infra#188)

* Update typed-store version to v0.4.0 (MystenLabs/mysten-infra#193)

* chore: update fastcrypto (MystenLabs/mysten-infra#194)

* codec: use bincode::deserialize instead of deserialize_from (MystenLabs/mysten-infra#195)

* Update FastCrypto (MystenLabs/mysten-infra#196)

* feat: add MallocSizeOf for OnceCell (MystenLabs/mysten-infra#197)

* chore(deps): remove unused dep (MystenLabs/mysten-infra#199)

* chore: update fastcrypto (MystenLabs/mysten-infra#198)

* Merge conflicts

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Brandon Williams <brandon@mystenlabs.com>
Co-authored-by: François Garillot <francois@garillot.net>
Co-authored-by: Mark Logan <103447440+mystenmark@users.noreply.github.com>
Co-authored-by: Anastasios Kichidis <akihidis@gmail.com>
Co-authored-by: Evan Chan <velvia@gmail.com>
Co-authored-by: laura <51134415+laura-makdah@users.noreply.github.com>
Co-authored-by: ade <93547199+oxade@users.noreply.github.com>
Co-authored-by: punwai <pun@mystenlabs.com>
Co-authored-by: mwtian <81660174+mwtian@users.noreply.github.com>
Co-authored-by: François Garillot <4142+huitseeker@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Lu Zhang <8418040+longbowlu@users.noreply.github.com>
Co-authored-by: aschran <aschran@mystenlabs.com>
Co-authored-by: Andrey Chursin <andll@danasoft.ws>
Co-authored-by: Ian Macalinao <github-primary@igm.pub>
Co-authored-by: Erwan <7871622+erwanor@users.noreply.github.com>
Co-authored-by: benr-ml <112846738+benr-ml@users.noreply.github.com>
Co-authored-by: Alberto Sonnino <alberto@sonnino.com>
Co-authored-by: brad-mysten <107816765+brad-mysten@users.noreply.github.com>
Co-authored-by: Joy Wang <108701016+joyqvq@users.noreply.github.com>
Co-authored-by: Sam Blackshear <sam@mystenlabs.com>
Co-authored-by: Brandon Williams <bwilliams.eng@gmail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@oxade oxade oxade left review comments

@huitseeker huitseeker huitseeker left review comments

@lanvidr lanvidr lanvidr approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@aschran @bmwill @huitseeker @lanvidr @oxade