Proposal: Configurable SVID rotation threshold on Agents

[hiyosi]

This is the issue that previously discussed (#1754) .
Previous issue was closed, however I still need the feature.

Proposal

The SVID rotation period is configurable.

Motivation

In my production, There are different implementations of SPIFFE are running.
(e.g. SPIRE and Athenz)

It is desirable that SVIDs issued by any implementation have the same availability level.
(e.g. SVID TTL=7days, Rotation=everyday)

Details

A parameter is similar to the cert-manager.

agent {
    experimental {
      rotation_config {
        x509_rotate_before = "144h"
      }
    }
}

Above example means that the Agent will rotate the SVID 6 days before expiration.

Limitations:

• The lower limit is considered to be a value such as 1m.
• There is no upper limit, If the x509_rotate_before is grater than the SVID TTL, the Agent will fallback to default behavior (rotation at 1/2 of TTL).

I have a PoC code, so please refer to check more details.
#4116

[evan2645]

Hi @hiyosi , thank you for opening this.

I feel that trying to align rotation semantics across different SPIFFE implementations is a bit of a moving target and probably not practically achievable in the long run ... however, I also understand that having control over these rotation semantics allows you to more tightly control you availability/security trade-offs

We discussed this issue in a maintainer call last week, and the biggest concern is around usability of this feature, as well as the impact. We agreed that we should probably not subject signing keys to configuration in this way ... the logic there is more nuanced. We also agreed that this agent configurable should apply equally to agent SVIDs, workload SVIDs, and X509 and JWT. What do you think?

In terms of usability, having another duration-based configurable feels unwieldy. Questions around the min/max come up (as you have pointed out), and it impacts SVIDs with different TTLs differently. What happens when an SVID lifetime just barely exceeds the x509_rotate_before value? etc ... Two alternative ideas were discussed:

• Rotation levels or modes: we could have something like "lazy", "default", and "aggressive" to describe how quickly SVIDs will be rotated, as a function of their relative lifetime. The advantage of this approach is that it's easy to understand and think about, and we can easily change the internal details of what they mean in the future.
• Rotation configurable as a fraction of TTL: we could configure a value between 0-1, indicating at what point the agent should begin to attempt rotation, expressed as a fraction of the TTL of the SVID in question. The default can be 0.5 which is the current behavior. The advantage of this approach is that it gives more control, but at the cost introducing the potential for a bad experience (e.g. if a user configures 0.99 and normally has large TTL values, but then sets agent_svid_ttl to a low number).

[hiyosi]

@evan2645
I appreciate the discussion.

I feel that trying to align rotation semantics across different SPIFFE implementations is a bit of a moving target and probably not practically achievable in the long run .

My motivation is to provide users with the same availability level of SVIDs.
So, the objective may be different than in the past issue.

probably not practically achievable in the long run

I'm no sure why?
(sorry, maybe I didn't understand correctly)

We also agreed that this agent configurable should apply equally to agent SVIDs, workload SVIDs, and X509 and JWT. What do you think?

Should the rotation configurable be the same even though the ttl configurable are different?
X.509 and JWT have different use-cases and different TTL sizes, so I think one value would not work well even if the configurable is like ratio.

e.g. The impact of 5% for 24h is different from 5% for 5m.

So, It would be better to be able to configure JWT and X.509 for different purposes.

Rotation levels or modes:

I think it is difficult to understand when the SVID will be rotated.

Rotation configurable as a fraction of TTL:

Actually, I considered this approach before making this proposal.
In my opinion, there are concerns about it need for calculations and indivisible values for specific value. (e.g. 0.3333....).
However, we can achieve something close to my purpose by this idea as well.

Honestly, I think it is a matter of preference which is easier to understand.

[hiyosi]

What should I do next?

[evan2645]

probably not practically achievable in the long run
I'm no sure why?
(sorry, maybe I didn't understand correctly)

Just because we have no control over what rotation semantics other SPIFFE implementations choose to implement, and it's hard to see how SPIRE can support all the different possibilities as the number of SPIFFE implementations grows.

Should the rotation configurable be the same even though the ttl configurable are different?
X.509 and JWT have different use-cases and different TTL sizes, so I think one value would not work well even if the configurable is like ratio.

Hmm.. yes, maybe worth some more thought. Ultimately just trying to figure out how we can support your request without creating an overly complicated solution that is difficult to configure or easy to do the wrong thing. Perhaps two configurables isn't bad, and I think there is precedent for it. On the other hand, it might not be clear that something like x509_rotate_before also affects agent availability

Rotation levels or modes:
I think it is difficult to understand when the SVID will be rotated.

It could be easier to understand if they are implemented as a fraction, and we can document it. I agree though that it is less clear than a straight number.

Actually, I considered this approach before making this proposal.
In my opinion, there are concerns about it need for calculations and indivisible values for specific value. (e.g. 0.3333....).

Yes that is one down side to the approach ... but IMO the downside with a static duration (e.g. not rotating until minutes/seconds prior to expiry) is more severe

What should I do next?

We discussed this issue again on Tuesday, and neither of the solutions proposed thus far are feeling like a great fit. During our discussion, we talked about what it is that users are really trying to achieve with this configurable, and the conclusion was that we're trying to achieve some kind of manageable floor for availability purposes. There is an additional aspect which is security related, but that can be controlled by existing TTL configurables.

The outcome of the discussion was that being able to simply configure SPIRE with your desired level of availability is probably a good idea, and makes the functionality clear. I think that ends up being close to what you originally proposed, but with some added semantics and a different name.

agent {
     availability_target = 7d
}

When set:

• SVIDs will never rotate prior to ~20% of their lifetime (to prevent tight renewal loops where SVID TTL is just slightly higher than the availability target)
• SVIDs older than ~20% of their lifetime, but with less than the availability target remaining, will be rotated
• SVIDs older than ~50% of their lifetime will be rotated regardless of the availability target (to prevent low availability target from triggering last minute renewal on longer TTLs)
• If the agent caches a new entry that has a TTL set wherein it is not possible to reach the availability target, a warning is logged

I just took a guess at the percentages, we can tweak them however we feel is appropriate .. I think the more important bit is how we react in the various situations. What do you think?

[hiyosi]

Just because we have no control over what rotation semantics other SPIFFE implementations choose to implement, and it's hard to see how SPIRE can support all the different possibilities as the number of SPIFFE implementations grows.

I understood. I may have made a bad(misleading) proposal.

I just took a guess at the percentages, we can tweak them however we feel is appropriate .. I think the more important bit is how we react in the various situations. What do you think?

It looks fine!
Ideally, I would be happy if there is a way to relax the restrictions with an understanding of the risks.
e.g. parameter with prefix subtle_*, unsafe_*.

Some my environments run with large TTL and short rotation period.
e.g. ttl=10d, availability_target=1d, ttl=30d, availability_target=1d

[evan2645]

It's still not clear to me if something like availability_target exactly solves your use case. The TTL examples you provided are helpful, perhaps we can use those as an example?

When I consider availability_target, the meaning is "here is the amount of time I want my deployment to survive in the event of a server/rotation outage". Since this is currently hardcoded to 50% of an SVIDs TTL, my assumption is that people setting the availability_target will be asking us to rotate more aggressively than 50%, such that a longer window of availability is maintained.

However, in your example ttl=10d, availability_target=1d, I think the result is actually less? Is it that you want to a 10 day TTL and you want to rotate every day? Just trying to make sure we're doing the right thing here

[hiyosi]

@evan2645

Is it that you want to a 10 day TTL and you want to rotate every day?

That’s correct.
So is the above configuration like this ?
ttl=10d, availability_target=9d

[hiyosi]

any updates?