Refactoring system interface #15

raoulstrackx · 2020-10-13T12:29:16Z

With the arrival of SGXv2, enclaves can dynamically add and remove memory pages. In SGXv1 this was not supported. Instead of adding the required changes for SGXv2 to this crate, the choice was made to factor out the sys interface. Now users of dlmalloc can simply implement the System trait. This allows further changes to this trait implementation (e.g., for SGXv2 support) without impacting this crate.

cc: @jethrogb

src/dlmalloc.rs

jethrogb

In addition, please change all existing platforms to the new API.

src/lib.rs

src/dlmalloc.rs

src/lib.rs

src/dlmalloc.rs

tests/smoke.rs

src/global.rs

src/lib.rs

src/global.rs

src/dlmalloc.rs

alexcrichton · 2020-10-20T21:54:11Z

Thanks for the PR! I'm not sure though that this is something I want to commit to in this crate. I'd prefer to not have a trait that exposes the underlying required operations since that's difficult to change over time. If sgx is changing rapidly then it may be best for a separate allocator other than this crate (although maybe based on this one?) to be used in libstd.

raoulstrackx · 2020-10-21T09:07:44Z

The main purpose of the PR is not because SGX may change (again) in the future. It's because there is a dependency inversion; implementing the system allocator for SGX (and other targets) relies on information provided by libstd.
Going over all commits since the initial release of the crate, none of them changed the System interface. I think that is as expected as the crate is based on the dlmalloc C implementation that in itself is also very stable.

alexcrichton

Sorry it's been a hectic week, but that makes sense if the embedding has application-specific knowledge about how to implement the allocator. I've left some comments throughout.

src/dlmalloc.rs

src/lib.rs

alexcrichton · 2020-10-28T15:13:56Z

src/lib.rs

+/// Instances of this type are used to allocate blocks of memory. For best
+/// results only use one of these. Currently doesn't implement `Drop` to release
+/// lingering memory back to the OS. That may happen eventually though!
+#[cfg(any(target_os = "linux", target_arch = "wasm32", target_os = "macos"))]


This is similar to my comment above, but I would prefer that this structure not have a conditional definition.

It only conditionally sets the type parameter default. I don't see how you could have a default (in order not to break existing users) but also have platforms that don't have an in-crate implementation without conditional compilation.

Platforms which don't have an implementation in this crate would default to just saying "allocation failed"

I think this #[cfg] can be removed?

tests/smoke.rs

raoulstrackx · 2020-10-28T16:12:10Z

No worries about the timing @alexcrichton, thanks for changing your point of view on this. I'll address your comments as soon as possible.

src/dlmalloc.rs

src/lib.rs

alexcrichton · 2020-10-29T16:01:18Z

src/lib.rs

+/// Instances of this type are used to allocate blocks of memory. For best
+/// results only use one of these. Currently doesn't implement `Drop` to release
+/// lingering memory back to the OS. That may happen eventually though!
+#[cfg(any(target_os = "linux", target_arch = "wasm32", target_os = "macos"))]


Platforms which don't have an implementation in this crate would default to just saying "allocation failed"

raoulstrackx · 2020-10-30T08:27:42Z

I don't see what exactly a default implementation of System that always returns an error would contribute. Isn't this just replacing a compile-time error with a runtime error? Agreed removing the conditional compilation would make the code more readable, but currently that is contained within the struct Dlmalloc. Note that such conditional compilation already needs to exist to select the correct sys module.

alexcrichton

I personally find that dealing with #[cfg] soup is much less fun and flexible than having APIs available that simply return errors. It's not great if you're not expecting an error but the error comes out really fast so long as you use basically anything. Overall I think the user experience is improved with fallible runtime methods rather than not-provided-at-compile-time methods.

src/dlmalloc.rs

src/lib.rs

tests/smoke.rs

src/lib.rs

alexcrichton · 2020-11-02T16:07:30Z

src/lib.rs

+/// Instances of this type are used to allocate blocks of memory. For best
+/// results only use one of these. Currently doesn't implement `Drop` to release
+/// lingering memory back to the OS. That may happen eventually though!
+#[cfg(any(target_os = "linux", target_arch = "wasm32", target_os = "macos"))]


I think this #[cfg] can be removed?

src/lib.rs

alexcrichton · 2020-11-02T16:08:44Z

src/lib.rs

+
+impl<S> Dlmalloc<S> {
+    /// Creates a new instance of an allocator
+    pub const fn new_with_platform(sys_allocator: S) -> Dlmalloc<S> {


I think this may want to be new_with_system since the name of the trait is "System"?

This can also be lumped in with the impl block below I think.

Unfortunately, that results in trait bounds other than Sized on const fn parameters are unstable. I'm assuming you'd want to avoid a const_fn crate attribute?

src/dlmalloc.rs

src/lib.rs

alexcrichton · 2020-11-02T16:14:33Z

src/lib.rs

@@ -29,15 +30,54 @@ mod dlmalloc;
 #[cfg(all(feature = "global", not(test)))]
 mod global;

+/// A platform interface
+pub unsafe trait System: Send {


To bikeshed a bit, the documentation and wording here are used somewhat interchangeably (e.g. this trait's documentation says "platform" where the trait says "system"). Perhaps the trait could be Allocator and the default struct could be System (to mirror std::alloc a bit).

src/lib.rs

alexcrichton · 2020-11-04T15:52:24Z

Ok this looks good to me, but I think CI is failing?

jethrogb · 2020-11-04T16:30:22Z

CI is failing on master too

alexcrichton · 2020-11-06T15:05:33Z

Thanks again!

raoulstrackx · 2020-11-06T15:10:37Z

Thanks for merging!

…Simulacrum Upgrade dlmalloc to version 0.2 In preparation of adding dynamic memory management support for SGXv2-enabled platforms, the dlmalloc crate has been refactored. More specifically, support has been added to implement platform specification outside of the dlmalloc crate. (see alexcrichton/dlmalloc-rs#15) This PR upgrades dlmalloc to version 0.2 for the `wasm` and `sgx` targets. As the dlmalloc changes have received a positive review, but have not been merged yet, this PR contains a commit to prevent tidy from aborting CI prematurely. cc: `@jethrogb`

jethrogb reviewed Oct 13, 2020

View reviewed changes

src/dlmalloc.rs Outdated Show resolved Hide resolved

jethrogb reviewed Oct 13, 2020

View reviewed changes

src/lib.rs Outdated Show resolved Hide resolved

src/dlmalloc.rs Outdated Show resolved Hide resolved

src/dlmalloc.rs Outdated Show resolved Hide resolved

src/lib.rs Outdated Show resolved Hide resolved

raoulstrackx force-pushed the raoul/interface branch 2 times, most recently from 432a542 to 356d1ea Compare October 14, 2020 14:30

jethrogb reviewed Oct 14, 2020

View reviewed changes

raoulstrackx force-pushed the raoul/interface branch from 356d1ea to d7f5a05 Compare October 14, 2020 15:49

jethrogb reviewed Oct 14, 2020

View reviewed changes

src/lib.rs Outdated Show resolved Hide resolved

src/lib.rs Outdated Show resolved Hide resolved

src/global.rs Outdated Show resolved Hide resolved

raoulstrackx force-pushed the raoul/interface branch from d7f5a05 to d69304d Compare October 15, 2020 09:11

jethrogb reviewed Oct 15, 2020

View reviewed changes

src/dlmalloc.rs Outdated Show resolved Hide resolved

raoulstrackx force-pushed the raoul/interface branch from d69304d to e3e1305 Compare October 15, 2020 10:20

jethrogb approved these changes Oct 15, 2020

View reviewed changes

raoulstrackx mentioned this pull request Oct 20, 2020

Upgrade dlmalloc to version 0.2 rust-lang/rust#78138

Merged

alexcrichton reviewed Oct 28, 2020

View reviewed changes

alexcrichton reviewed Oct 29, 2020

View reviewed changes

alexcrichton reviewed Oct 30, 2020

View reviewed changes

src/dlmalloc.rs Outdated Show resolved Hide resolved

src/lib.rs Outdated Show resolved Hide resolved

tests/smoke.rs Outdated Show resolved Hide resolved

raoulstrackx force-pushed the raoul/interface branch 2 times, most recently from 8b0a8ee to c798baf Compare November 2, 2020 09:21

alexcrichton reviewed Nov 2, 2020

View reviewed changes

raoulstrackx force-pushed the raoul/interface branch from 367bab4 to 97ff3f5 Compare November 3, 2020 14:40

alexcrichton reviewed Nov 4, 2020

View reviewed changes

src/lib.rs Outdated Show resolved Hide resolved

src/lib.rs Outdated Show resolved Hide resolved

src/lib.rs Outdated Show resolved Hide resolved

raoulstrackx added 4 commits November 5, 2020 10:05

Fixing allocator-api feature

622ff22

Moving sys modules behind a trait

1ff455d

Implementing Linux system Trait

f0af0d3

Implementing MacOS System trait

c2f3af6

raoulstrackx added 6 commits November 5, 2020 10:09

Implementing wasm System trait

855c93d

Implementing Sgx system trait

3704f05

removing SGX platform

1de22ae

Added default type parameter

b658008

Bumped version

0272ed2

Addressing reviewer's comments

b99a307

raoulstrackx force-pushed the raoul/interface branch 2 times, most recently from 944418c to b1f9c2f Compare November 5, 2020 15:55

Removing allocator-api support

4786050

raoulstrackx force-pushed the raoul/interface branch from b1f9c2f to 4786050 Compare November 5, 2020 16:22

raoulstrackx mentioned this pull request Nov 5, 2020

Fixing allocator-api feature #16

Closed

alexcrichton merged commit fbad771 into alexcrichton:master Nov 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactoring system interface #15

Refactoring system interface #15

raoulstrackx commented Oct 13, 2020

jethrogb left a comment

alexcrichton commented Oct 20, 2020

raoulstrackx commented Oct 21, 2020

alexcrichton left a comment

alexcrichton Oct 28, 2020

jethrogb Oct 29, 2020

alexcrichton Oct 29, 2020

alexcrichton Nov 2, 2020

raoulstrackx commented Oct 28, 2020

alexcrichton Oct 29, 2020

raoulstrackx commented Oct 30, 2020

alexcrichton left a comment

alexcrichton Nov 2, 2020

alexcrichton Nov 2, 2020

alexcrichton Nov 2, 2020

raoulstrackx Nov 3, 2020

alexcrichton Nov 2, 2020

alexcrichton commented Nov 4, 2020

jethrogb commented Nov 4, 2020

alexcrichton commented Nov 6, 2020

raoulstrackx commented Nov 6, 2020

Refactoring system interface #15

Refactoring system interface #15

Conversation

raoulstrackx commented Oct 13, 2020

jethrogb left a comment

Choose a reason for hiding this comment

alexcrichton commented Oct 20, 2020

raoulstrackx commented Oct 21, 2020

alexcrichton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raoulstrackx commented Oct 28, 2020

Choose a reason for hiding this comment

raoulstrackx commented Oct 30, 2020

alexcrichton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Nov 4, 2020

jethrogb commented Nov 4, 2020

alexcrichton commented Nov 6, 2020

raoulstrackx commented Nov 6, 2020