Disallow mixed device type on the same pool?

[leseb]

In the hope to bring a good user experience, I'm wondering if Stratis should refuse to create a pool with mixed-type devices. For instance a pool with HDD and SSD. It's probably best to have an SSD pool and an HDD pool instead of mixing both. Performance is the main driver for this request, it would be strange to have a pool with sporadic performance depending on which blockdev the fs ends up being created. One thing that we could benefit from is adding a 'type' or a 'class' to a pool to identify its performance characteristics or guess at a glance.

Thoughts?

[mulkieran]

We have a facility for scrutinizing combinations of blockdevs at pool creation, cache initialization, and block device addition. At this time, we only check for compatible sector sizes of individual block devices and we make these checks to prevent actual failure of the pool, rather than to enforce consistency of performance on the pool.

We would not be averse to expanding this facility in the way you suggest.

But to enforce what would be a policy we need to know the user's intent. We could, for example, rather than discovering the characteristics of a pool, ask the user to specify their intent, and make sure that the pool is constructed with devices that match that intent.

Another angle is the questions of the cache. It would be foolish for a user to use, for their cache, devices that would prove slower than those in their data tier. At present, we do not enforce any restriction on the devices in the cache tier vis-a-vis the data tier, but that might be the most obvious place to start, when considering policy.

[leseb]

Thanks for your reply.

Why would a user have to provide an intent that they are looking for an optimal configuration? As a Stratis user, this is the type of decision that I expect Stratis to make for me. Since usability and ease of use are key to Stratis I'd argue that Stratis should at least warn the user. For instance, I can imagine a situation where the pool is entirely SSD based and the admin attempts to add an HDD. Ideally, Stratis would warn or reject and the admin would need to pass a specific flag to force that configuration anyway. Better be safe than sorry, once there is data on the fs, it's too late.

The cache is a good example and just categorizing this action as "foolish" does not help. We cannot expect users to know everything, we need to guide them and hide the complexity. It's always best to expect nothing from an end user and assist as much as possible.

Perhaps we are saying the same thing and it's how we present it where we diverge, but the example I gave seems so trivial and the benefits are clearly identifiable that the policy seems a bit over-engineered at this point.

Happy to discuss more :).

[mulkieran]

I do think you may be almost completely reversing my meaning. Here's what I'm trying to say.

In Stratis, we do not have a --force flag, i.e., a generic option that allows a user to override a restriction. Thus, we have to be careful to avoid establishing and enforcing policy that might not actually be the optimum policy for some users, as they will likely not be able to override the policy.

When it comes to the cache, it just makes no sense to make a cache that is bound to be slower than its data tier. So, this is a particularly obvious situation where we could fairly confidently establish a policy that we then enforce without any loopholes or escape hatches for the users.

Your SSD and HDD example is not quite as obvious as the cache example, because if a user add the SSD device to the hypothetical HDD pool they have gained something, i.e., more storage space (even though the pool has also gained weird performance characteristics). So, we have to do more subtle thinking about what our policy should be in that case and I don't presently have a definite answer about what would be best.

One of our users brought up an interesting question[1] about how we configure our dm-crypt devices which is far more subtle. It's clear that the configuration can have an effect, but it's not at all clear that there is one best choice. So that's harder still.

[1] #521

[leseb]

I do think you may be almost completely reversing my meaning. Here's what I'm trying to say.

That was not my intention.

In Stratis, we do not have a --force flag, i.e., a generic option that allows a user to override a restriction. Thus, we have to be careful to avoid establishing and enforcing policy that might not actually be the optimum policy for some users, as they will likely not be able to override the policy.

Hence the proposal of a '--force' flag so the user can override the policy if not suitable.

When it comes to the cache, it just makes no sense to make a cache that is bound to be slower than its data tier. So, this is a particularly obvious situation where we could fairly confidently establish a policy that we then enforce without any loopholes or escape hatches for the users.

Agreed this one is obvious. Do you want to tackle it first?

Your SSD and HDD example is not quite as obvious as the cache example, because if a user add the SSD device to the hypothetical HDD pool they have gained something, i.e., more storage space (even though the pool has also gained weird performance characteristics). So, we have to do more subtle thinking about what our policy should be in that case and I don't presently have a definite answer about what would be best.

How about a warning printed for now then? So at least the user realizes and we can avoid mistakes that way.
How do you propose we do that more subtle thinking then? Meeting?

One of our users brought up an interesting question[1] about how we configure our dm-crypt devices which is far more subtle. It's clear that the configuration can have an effect, but it's not at all clear that there is one best choice. So that's harder still.

[1] #521

Yes, I've seen that issue before, Ceph engineering actually requested the upstream Kernel commits to be backported to RHEL. This code is available in cryptsetup-2.3.7.
Yes, this one is harder and much more subtle than my initial request about mixed devices in the same pool. However, projects wrapping on top of existing technologies should make an opinionated decision after careful consideration, especially since those flags don't seem to be enabled by default in cryptsetup...