Make remote state backends a pluggable option

[nickithewatt]

As with the current pluggable terraform provider approach, it would be very handy to have the ability to plug in custom remote state backends without having to have them directly incorporated into the core terraform code.

This would allow people to develop and make use of a variety of different remote stores such as custom Databases etc in an independent manner, yet still allow for it to be contributed back if more widely available.

[jen20]

Hi @nickithewatt! This is not something we have previously considered as far as I am aware, and is not currently on the roadmap. I think in the short term this is something we are unlikely to prioritise internally, and it's hard to justify separate binaries at this stage. That said, the new go-plugin system may allow for us to have internal plugins while still also allowing external ones, so we may come back to this at some point in future.

[nickithewatt]

Thanks for the reply @jen20, I understand and to be fair its not causing me massive grief at present, but when I started looking into this thought it would be a nice approach. I'll keep an eye on it anyway, and watch out for possible go-plugin niceness :) thanks!
Also happy Easter BTW

[rremer]

I think there's additional benefit to a provider-approach for backends which is the separation of bug triage. Currently there are over 200 bugs in this repo related to backends, which could probably be better served by specific maintainers.

Probably not worth doing before 0.12 is done.

[schrodervictor]

I also support this idea. Here, for example, we use a GPG based approach for other repositories to encrypt secrets, commit them to the code base and and, at the same time, have consistent credentials rotation and access control across the team members.

We would like to combine this solid strategy with the remote storage of the state on S3, for example, so we don't have to care about the secrets being present in the state file. Unfortunately, there isn't any hook in terraform allowing us to do that.

I can clearly imagine how to implement such strategy as a plugin, but I guess the only solution right now would be to add it to the core as a new backend (or configuration to any backend). Would the core maintainers consider accepting a pull request in this direction?

[kshpytsya]

@schrodervictor How about using the existing http backend?

[schrodervictor]

@kshpytsya I'm aware of this possibility, but unless I'm not seeing the full picture, I would be obliged to keep a service running and reachable to provide states and locks. That's cool and gives more control, but doesn't solve the problem of having encryption of data at rest, because I would need to give the server -- somehow -- the knowledge to decrypt the data. I want the decryption to happen on the client side.

I'm also aware that S3 does provide encryption for data at rest (in combination with KMS) and in transit (using https transport) but the proposal is to comply with zero-knowledge requirement on the server, which is not possible on both backends, as Terraform doesn't offer a hook to run decryption (possibly with interactive tty for pinentry), after a encrypt file is received from an endpoint (S3 or custom http).

Do you think it's possible with the current backends? If at least I could point to a local executable file as a backend...

[kshpytsya]

@schrodervictor I was thinking more in vein of having a wrapper around your terraform executions that would bring up a local server, which you could implement in whatever language you are comfortable with. So you could consider this server to be on the client side. Since AFAIK terraform does not yet provide automated means of downloading third-party providers, I would argue that logics-wise this solution, while requiring extra effort over plain terraform, would not be more complex than having to deploy your custom plugin.

[schrodervictor]

@kshpytsya I haven't considered to have the http server running locally. Yes, that could work and this idea would solve the problem of having decryption happening on the client side, but it's a reeeeeeally dirty workaround... 😒

[rremer]

I had this same thought and threw it out because the http server would need to be started out of band, e.g. there's no guarantee it's running because it's not a part of the terraform graph.

If you were to run your own http service locally, you might as well run consul or minio locally and point their storage at your local filesystem. This would not improve the productivity of the team :p