LXD backend discovery support

[jtopjian]

Hello,

I think gobetween would be a great fit for LXD containers, so I made a proof-of-concept discovery. The idea is to configure the LXD discovery like any other discovery mechanism, and then launch an LXD container like so:

$ lxc launch ubuntu foo --config user.gobetween.label="foo" --config user.gobetween.private_port=80

LXD reserves the user.* config key for user-specified metadata. The full config reference is here

Right now, discovery is only done on the local server, so only local containers are discovered. However, it could be extended to support remote LXD server(s).

I'm happy to answer any questions about this feature. In addition, I wasn't able to find a doc that lists the requirements for contributing a patch, so please let me know if I need to do anything else.

What I would really like to do is use gobetween to dynamically generate server entries based on containers. This is because LXD does not provide a built-in mechanism to automatically handle external-to-container traffic (similar to publishing a docker port). In effect, gobetween would supply that support. I understand this might be too much of a niche case for gobetween, though. Perhaps a better solution is to write a glue utility that bridges the LXD server and gobetween by using the gobetween REST API rather than have gobetween directly support this.

[illarion]

Your pull request is simple and awesome, good work! Could you please provide me some instructions on how to setup test environment, in order to try it? I am not familiar with lxc/lxd.

Regarding requirements for contributing a patch, we don't have such requirements yet, but it seems that we have to write them down, so, for now, in this particular case, it would be nice to have support not only for local containers, but also for remote ones.

Thank you!

[yyyar]

Hi @jtopjian! Thanks for the pull request, it looks really good! We had plans to add lxd support some time ago but never had enough time to implement it.

It would be cool to have all types of containers (local/remote) support as @illarion mentioned, and also configurable variables (config) names, i.e. make "user.gobetween.label="foo" configurable in gobetween discovery config to make it more flexible, something like this (maybe makes sense to name it 'metadata', to be more close to lxc API terminology):

  LXDContainerMetadataKey       string `toml:"lxd_container_metadata_key" json:"lxd_container_metadata_key"`
  LXDContainerMetadataValue       string `toml:"lxd_container_metadata_value" json:"lxd_container_metadata_value"`

So one can use:

lxc launch ubuntu foo  --config user.any.key="any.value" ...

Also I noticed you declared LXDContainerPrivatePort and LXDContainerInterface in config but do not use it in discovery implementation.

Please provide some instructions how to setup simple test environment so we can try/test and give you more feedback.

Thanks!

[jtopjian]

Hello,

Could you please provide me some instructions on how to setup test environment, in order to try it?

Sure. It's easiest to use an Ubuntu-based distribution, notably 16.04. You can use any kind of virtual machine. Run the following commands:

$ apt-get update
$ apt-get lxd
$ lxd init  (Just use the default answers for all questions)
$ lxc image copy images:ubuntu/xenial/amd64 local: --alias ubuntu
$ lxc launch ubuntu foo --config user.gobetween.label="foo" --config user.gobetween.private_port=80
$ lxc list
$ lxc exec foo /bin/bash

You can then run gobetween on the same vm and have it detect the running container.

Edit: Also, this blog series covers LXD in depth if you'd like to learn more.

it would be nice to have support not only for local containers, but also for remote ones.

Yes, I can get this implemented. It's quite simple to add support for a remote LXD server that a client (in this case, the gobetween server) has previously authenticated with. This is done by installing the lxc client on the server and then doing:

$ lxc remote add lxd-server lxd-server.example.com

But I recently did work for a Terraform plugin that does all of the remote work without the need of installing the client:

terraform-lxd/terraform-provider-lxd@71d7876

terraform-lxd/terraform-provider-lxd@40fd7e3

It'd be quite simple to copy that work over, but just to give an explanation of why there's so much code :)

also configurable variables (config) names, i.e. make "user.gobetween.label="foo" configurable in gobetween discovery config

👍

Also I noticed you declared LXDContainerPrivatePort and LXDContainerInterface in config but do not use it in discovery implementation.

Yes, you're right. I'll get that fixed up.

Thank you both. Let me know if you have any other feedback or questions :)

[illarion]

Still lacks remote containers support, but very clean and good code

[illarion]

What is changed here?

[jtopjian]

My vim config automatically runs gofmt upon save. It must have done some tabs/space fix.

[illarion]

You can move defaults initialization to manager.go

[jtopjian]

👍

[jtopjian]

Still lacks remote containers support, but very clean and good code

Thanks! Yes, I apologize, I'm going to be working on remote support over the next couple of days. I will mark this PR as WIP until it's ready to go.

[jtopjian]

Connecting to a single remote LXD server is now possible. As you can see, it's much more complicated than connecting to an unauthenticated local socket, so no rush to review and let me know if you have any questions. I've tried to make helpful comments in the code to explain what's going on.

Briefly, there are 3 ways to connect to an LXD server:

1. Locally, via socket
2. Remotely via PKI. This means that you have a local Certificate Authority and have pre-provisioned the server and client with a cert, key, and CA cert. If a CA cert is detected on the server, no certificate exchange is done with the client. The client must still authenticate with the server once, though. Think of this as a one-time HTTP basic-auth over SSL.
3. Remotely via certificate exchange. This is when both the client and server generate their own certificates and exchange them through implied trust. This is why I added an option called lxd_client_accept_server_cert so the user must explicitly say this is OK. Authentication with a password must still be done.

With 2 and 3, the LXD client stores its client certificates and server certificates in the "config directory". By default this is ~/.config/lxd, but I made this a configurable option in case someone wanted to store certificates elsewhere.

This PR supports all 3 methods.

One thing to keep in mind is that, as I mentioned in the beginning, LXD has no way of natively forwarding traffic from the host to the container the way Docker does with publishing ports. Because of this limitation, connecting to remote LXD servers only makes sense in two scenarios:

1. Some other port-forwarding configuration has been done.
2. The containers were launched on a bridged network that is shared with gobetween.

Let me know if you have any questions about anything at all 😄

If this all looks good, I can look into support for multiple LXD servers, though perhaps that should be a separate PR to keep review manageable?

Thinking out loud: Have you guys thought about making gobetween itself a discovery mechanism? Let's say I use gobetween locally on a bunch of LXD servers. The local gobetween daemons handle forwarding from the LXD host to the LXD containers. Then I create an upstream gobetween load balancer that discovers services by connecting to the downstream gobetween daemons via the REST API.

[yyyar]

Hi @jtopjian, thanks for the update!
Just to give you a quick feedback, we're looking on your changes and everything looks good. But it may take a bit longer for us to try all these cases. I've managed to try local case and it works fine, however there are some things I think may be improved.

• There is iface global variable in lxd.go it may cause problems when using several different lxd discoveries (different servers) simultaneously. It's better to make it constant and copy when needed.
• Is cfg.LXDTimeout param really needed? Seems cfg.Timeout should be used for this, it's not global it's just per-discovery variable so it does the same job as cfg.LXDTimeout.
• Maybe move some client build / config preconfiguration out of lxdFetch to another function? It seems handling too much logic.

Please give us more time to test it :-)
Also, don't you mind if we do some commits to your pull request branch as well?

If this all looks good, I can look into support for multiple LXD servers, though perhaps that should be a separate PR to keep review manageable?

Yep I think it's better to finish with this PR first.

Thinking out loud: Have you guys thought about making gobetween itself a discovery mechanism? Let's say I use gobetween locally on a bunch of LXD servers. The local gobetween daemons handle forwarding from the LXD host to the LXD containers. Then I create an upstream gobetween load balancer that discovers services by connecting to the downstream gobetween daemons via the REST API.

We thought about something like this, but decided to postpone it until v1.0. We just want to make current features work well and release stable version and then think about further bigger changes. Anyways we're open for ideas and requests like this one, I think it worth to open another Issue concretely about this idea to discuss further.

Thanks!

[jtopjian]

Hi @yyyar,

There is iface global variable in lxd.go it may cause problems when using several different lxd discoveries (different servers) simultaneously. It's better to make it constant and copy when needed.

👍

Is cfg.LXDTimeout param really needed? Seems cfg.Timeout should be used for this, it's not global it's just per-discovery variable so it does the same job as cfg.LXDTimeout.

Perhaps I misunderstood, but I believe @illarion requested this above. I agree that it's kind of moot.

Maybe move some client build / config preconfiguration out of lxdFetch to another function? It seems handling too much logic.

Also, don't you mind if we do some commits to your pull request branch as well?

I agree about lxdFetch. It's a bit of a beast. I would be glad to try to simplify it. And you guys are more than welcome to branch, patch, modify, etc. Let me know if you want to try to work on simplifying lxdFetch, though, just so we don't step on toes 😄

Please give us more time to test it :-)

This is totally understandable. Please take your time.

Yep I think it's better to finish with this PR first.

👍

I think it worth to open another Issue concretely about this idea to discuss further.

👍

[jtopjian]

I agree about lxdFetch. It's a bit of a beast. I would be glad to try to simplify it.

I have some ideas for this. I'll take a stab at it over the weekend. :)

[jtopjian]

I pushed two new commits which should resolve some of the comments.

So should I remove lxdTimeout? Should lxdRetryWaitDuration stay?

[illarion]

@jtopjian I've reviewed the code again and would like to change it a bit, removing lxdTimeout lxdRetryWaitDuration and specifying defaults in example config. Please review the commit.

[jtopjian]

@illarion Looks good to me! :)

[shantanugadgil]

Awesome work with LXD as a first class back-end. Eagerly awaiting the release!
Nice work with documenting the planned features!! 🥇 ... gives a good feel for whats coming! 😄
BTW, any specific timeline for the merge to master?

[illarion]

@shantanugadgil we will perform last overview / check next week (Monday or Tuesday) and merge this.

[shantanugadgil]

@illarion any updates on the merge? - thanks!

[yyyar]

Hi Everyone!

Apologies for being unresponsive for so long. Finally I had a chance to spend some time looking into code again. I've tested it with local LXD and it works as expected. I've made some minor changes in the configuration and the code to make it align with other discovery types a bit better. I'll push changes it to this branch so you can review..

I know this PR is waiting to be merged for long time, but let me test remote https mode first, and I'll be happy to merge it. I plan to merge it to master till the end of next week.

[illarion]

I am now setting up test environment for remote mode..

[illarion]

@jtopjian there is one thing not related to the code, but to the setup, that I cannot understand:

I've set up lxd server onto separate machine, set up ubuntu container, running python -m http.server 80. This container has bridged network address, visible from within lxd host, but how should gobetween provide acces to it, if running not at the lxd host?

[jtopjian]

@illarion Yes, this is what I have been explaining above: since LXD does not provide a native way to publish ports, gobetween running on the LXD server takes over that responsibility. This alone is a good solution to a lot of people.

At the moment, the only things to do are:

1. Write iptables rules that forward traffic from the external address to the internal LXD container.
2. Create an LXD bridge that bridges your public interface (ie: eth0) to the LXD containers. This is dangerous if doing remotely as it can cut off all network access to your LXD server if you make a mistake. Your network also has to be able to support this type of bridged communication. For me, we do this on some bare metal servers, but if running in a cloud, traffic is usually restricted to only 1 MAC address unless a special configuration is made.

This is where my idea of chaining gobetween servers came from: have one running within the LXD server and another that is external.

[illarion]

@jtopjian this makes remote access from gobetween to lxd redundant for now. One can always set up gobetween on lxd host and make it discover backend containers locally, expose them to the public and then build any other infrastructure, even chain of gobetweens.

[illarion]

But ok, now I know that I should put gb to lxd host.

[jtopjian]

@illarion I agree completely.

[jtopjian]

@illarion Actually, I agree somewhat completely :)

this makes remote access from gobetween to lxd redundant for now

I would say this makes it redundant to most people for now. There are deployments where the containers are exposed on a publicly accessible interface. For these, gobetween does not need to run on the LXD server. But, it is safe to say that these deployments will be a small subset of the total usage.

[jtopjian]

@shantanugadgil If I understand you correctly, there are several listening services on each container and you want gobetween to be able to discover them all? If that is the case, at this time, gobetween will only detect one service/port per container. Multiple service/ports is certainly possible, but I think that's quite out of scope for an initial feature commit.

@illarion I was able to use macvtap. Communication between gobetween and the lxd container worked great. However, macvtap is susceptible to the same MAC address restrictions as running a bridge on the public interface in a cloud environment, though. In order to get around this, I set up the following environment:

In an OpenStack cloud, I launched two virtual machines, each on two networks:

network1: access to the public internet. this network has a mac address restriction where only recorded macs can communicate on the network. This is to prevent arp spoofing, attacking other customers, etc.

network2: private network with less restrictions.

vm1 ran gobetween

vm2 ran lxd. the lxd server ran on network1 and containers were launched on network2 using macvtap.

gobetween contacted lxd via network1 and received the backend information of containers running on network2. It was then able to direct traffic to network2 successfully.

In order to test a scenario where macvtap interfaces are on the same network as the listening lxd server, I'd need access to a different environment, which will take a few days.

Also, I noticed some of the configuration names were changed and weren't reflected in the config file template. I can submit a patch with these fixes.

[shantanugadgil]

I would recommend using simple full words "Cannot" instead of "Can't".

[shantanugadgil]

@illarion I think @jtopjian has confirmed the multiple services question, thanks.

@jtopjian yes, you are correct, there are multiple services inside a single LXD.
For an initial feature, a single service discovery is fine, I have switched back (for now) to my exec backend generating the multiple entries for me.
Considering LXD's are viewed more as lightweight VM's, multiple services would be a definite need, I think.

Also @illarion is talking about macvlan (and not macvtap), wonder if that matters to the discussion?

Thanks for the awesome work! 👍

Regards,
Shantanu

[jtopjian]

Also @illarion is talking about macvlan (and not macvtap), wonder if that matters to the discussion?

That was a typo on my part. I was using macvlan.

[jtopjian]

@jtopjian yes, you are correct, there are multiple services inside a single LXD.
For an initial feature, a single service discovery is fine, I have switched back (for now) to my exec backend generating the multiple entries for me.

I was thinking about this a little more and it seems like it could quickly become complicated. It's one thing to be able to provide an array of available ports, but (unless I'm not understanding what you're trying to do), providing an array of several configurations and then map them (ex: port-to-sni mapping) is susceptible to a lot of different issues, but in terms of user experience and configuration validation. I'm not saying it's not possible, but I have a feeling that you might be better off with a custom exec backend to provide your unique configuration.

[shantanugadgil]

I also thought about this.

In my mind, the user input should be some sort of map, rather than an array.
I think its easier to specify it as a map (only during input).
(The thought of this being deployed via TF)

While looking up the lxd config data, gobetween lxd discovery could lookup two arrays;
example:
user.gobetween.portN
user.gobetween.sniN

Other than, LGTM for now.

Thanks and Regards,
Shantanu

Edit: fixup the example (array reference) to what I really meant. 🤓

[yyyar]

There is one more thing

@jtopjian, you said you did the same thing in:

terraform-lxd/terraform-provider-lxd@71d7876
terraform-lxd/terraform-provider-lxd@40fd7e3

The code in the pull request is exact copy-paste of the code from commits above. That repository has Mozilla Public License 2.0, and I'm not sure if we can use the same code here without adding MPL. Sorry I've noticed it only now.

https://softwareengineering.stackexchange.com/questions/317944/can-i-include-code-licensed-under-mpl-within-a-project-under-mit-license

We'd like to keep everything MIT licensed.
Any ideas? :-)

As a last resort, we can keep/merge only local discovery for now, not including remote/https/client certificates generation code that is MPL licensed.

[illarion]

@jtopjian @yyyar there are 2 solutions I see for now, both with interactive rebase:

1. completely remove commits that have copy'n'pasted code, fix conflicts and leave local discovery only
2. do all stuff from 1 + implement remote support from scratch that is obviously more complex.

[jtopjian]

The code in the pull request is exact copy-paste of the code from commits above. That repository has Mozilla Public License 2.0, and I'm not sure if we can use the same code here without adding MPL.

This is interesting, notably because I'm the author of this. I am quite sure I have the right to be able to explicitly grant use of my code under a different license, but I will do some searching to confirm this. I'd be very surprised if I wasn't able to do this as the licenses would be dictating what I can and can't do with my own original works :)

[jtopjian]

Also, I suppose another option is for me to bundle the offending code into an external Go package and then call the exported method(s) from here. gobetween is using the Consul go package which is also under the Mozilla license, so that sets precedence for doing this.

[nickdoikov]

i`m not sure that you can recall your code or change license for it after merge into the project.

[jtopjian]

i`m not sure that you can recall your code or change license for it after merge into the project.

I'm not talking about changing the license of the submitted code. I'm saying that as the creator of said code, I should have the right to grant gobetween explicit permission to use the code that I created under an MIT license.

[jtopjian]

I've dug into this a little bit. First, I completely understand and appreciate the due diligence of ensuring software compatibility. I've had to do it a few times myself. So there's no hard feelings or offence. It's just not fun to read about licensing :)

Second, just to be entirely clear, the patches that were linked are now out of date. That code is no longer used. The updated implementations are here:

• https://github.com/sl1pm4t/terraform-provider-lxd/blob/ff6723b100fad7c9db0235e15c844e3ea6f2f506/lxd/provider.go#L212
• https://github.com/jtopjian/gobetween/blob/828807aba87b4e1fcf4bc3bb48d93325bd2144a4/src/discovery/lxd.go#L248

and

• https://github.com/sl1pm4t/terraform-provider-lxd/blob/ff6723b100fad7c9db0235e15c844e3ea6f2f506/lxd/provider.go#L232
• https://github.com/jtopjian/gobetween/blob/828807aba87b4e1fcf4bc3bb48d93325bd2144a4/src/discovery/lxd.go#L270

So there are two functions in question. It is important to note that I am the sole author of both functions. You can check the git blame report to verify.

From what I can find so far, these two posts may best describe this situation:

https://softwareengineering.stackexchange.com/questions/244520/who-owns-the-copyright-in-a-patch-file

https://softwareengineering.stackexchange.com/questions/347558/does-a-license-i-put-on-my-own-software-apply-to-me-can-i-change-the-license

I am submitting a patch to gobetween. The pieces of code in question were not modifying existing gobetween code, so there's not an issue of shared ownership here. The code in question is a standalone feature of my own original work. IMO, this grants me full ownership of this code.

Therefore, I'm in a position to explicitly grant permission to gobetween to use these pieces of code under an MIT license. I am happy to update the code and add an inline comment saying as much.

If the gobetween team is not comfortable with that, I can create a separate go package called github.com/jtopjian/lxdhelpers and add those two functions as exportable functions. That go package can then be added as a dependency to gobetween just like the other non-MIT dependencies.

Thoughts?

[jtopjian]

If the gobetween team is not comfortable with that, I can create a separate go package called github.com/jtopjian/lxdhelpers and add those two functions as exportable functions. That go package can then be added as a dependency to gobetween just like the other non-MIT dependencies.

Actually, I'll just go ahead and do this. In addition, I can alter the terraform-provider-lxd code to use the shared library. This should remove any concern about ownership and licenses.

I'll work on this shortly and update this code.

[nickdoikov]

Thanks, @jtopjian

All collaborators will discuss this tomorrow between us .
Please wait for a while.

[jtopjian]

All collaborators will discuss this tomorrow between us .
Please wait for a while.

Understood. However, I still think moving the code in question to a shared library is the best solution, so I have gone ahead and done it. It reduces duplicate code, thus reduces errors, and this issue of licensing/ownership should not come up again in another future project.

The same changes have been made in terraform-provider-lxd.

[yyyar]

@jtopjian
We did a bit more investigation on these license questions (https://opensource.guide/legal/) and as we understand you're right. Copyright holder (who is original work author) can do whatever he wants with his work, so you can copy your code here with different license (MIT). Sorry for the inconvenience. :-)

It's isn't necessary to move this code out in separate library as long as you grant permission for gobetween to include your code under MIT license , we can include it as source as it is. It's up to you.

So here is our action plan:
0. You have to decide whether you want to keep source and do it as separate lib. We're ok with both.

1. We fix missing build dependencies for Windows platform in master
2. We replace goxc cross-compilation with native gobuild with GOOS and GOARCH evn vars. This is because when we tried to build your PR, goxc failed to cross-compile lxc/lxd/shared for Windows. But native go build works good.
3. We merge your PR.

[jtopjian]

Sorry for the inconvenience. :-)

It's really no inconvenience at all. It's sometimes a necessary thing to do when accepting code contributions. I appreciate you bringing it up.

0. You have to decide whether you want to keep source and do it as separate lib. We're ok with both.

I'm going to leave those pieces of code as a separate library. This is for two reasons:

1. The code in this library can be applied to multiple projects. Even if I'm the only one using the library, it'll be easier for me to manage where I've added LXD support.

2. The code should probably be moved upstream to the LXD project itself. Those functions are really just workarounds for what should be upstream. The library singles out the needed functionality. I can work on this.

We fix missing build dependencies for Windows platform in master
We replace goxc cross-compilation with native gobuild with GOOS and GOARCH evn vars. This is because when we tried to build your PR, goxc failed to cross-compile lxc/lxd/shared for Windows. But native go build works good.

So now I feel bad that I've broke windows :)

Let me know if I can do anything to help with this.

[jtopjian]

🎉

Let me know if you guys have any questions or if I can help with documentation. I am also available to help support this after the 0.5.0 release.

[yyyar]

@jtopjian Thanks for awesome work 👍

Would be cool if you can help with brief documentation we can place in https://github.com/yyyar/gobetween/wiki/Discovery "lxd" section. You can do it here as markdown comment and then we'll move it to wiki.

[jtopjian]

Will do. Give me a day or so to write it up.

[jtopjian]

Here's a draft of the documentation. Please let me know if anything needs clarified. Also feel free to rewrite anything you see fit:

LXD

LXD discovery provides the ability to query an LXD server for hosted containers and then forward traffic to those containers.

There are two ways of using LXD discovery: running gobetween on the LXD server itself or having gobetween query a remote LXD server.

Local LXD Server

To have gobetween query a local LXD server, start with the following configuration:

[servers.example]

# ...

  [servers.example.discovery]
  kind = "lxd"
  lxd_server_address = "unix:///var/lib/lxd/unix.socket"
  lxd_container_label_key = "user.gobetween.label"
  lxd_container_port_key = "user.gobetween.port"
  lxd_container_label_value = "web"

Next, launch a container:

$ lxc launch <image_name> <container_name> -c user.gobetween.label=web -c user.gobetween.port=80

Finally, launch gobetween. gobetween will query the local LXD server for all containers with the metadata user.gobetween.label of web forward traffic to the port set in user.gobetween.port.

This can be an effective way of publishing local containers to the public interface of your LXD server.

Remote LXD Server

Querying a remote LXD server requires some extra configuration. First, you must make sure your LXD server is remotely accessible. See this blog post for instructions on how to do that.

Second, the server that will host gobetween must authenticate with the LXD server. The same blog post provides instructions on how to do that using the lxc client, but gobetween can also handle this internally, which means you do not have to install any LXC/LXD packages on the gobetween host.

To configure gobetween to query a remote LXD server, use the following configuration:

[servers.example]

# ...

  [servers.example.discovery]
  kind = "lxd"
  lxd_server_address = "https://lxd-01.example.com:8443"
  lxd_server_remote_name = "lxd-01"
  lxd_server_remote_password = "password"
  lxd_generate_client_certs = true
  lxd_accept_server_cert = true

  lxd_container_label_key = "user.gobetween.label"
  lxd_container_port_key = "user.gobetween.port"
  lxd_container_label_value = "web"

One important thing to keep in mind is that gobetween must be able to access the containers remotely. This means that your containers must be attached to a network accessible to gobetween or you have port forwarding rules in place to forward traffic to internally hosted containers.

Notes

• For the initial release of LXD support, only one LXD server per server entry is possible.
• By default, the container's eth0 interface is used. You can choose a different a different interface on your containers by either:
  1. Setting lxd_container_interface in gobetween.toml to an interface name (such as eth1) which will apply to all containers on the LXD server.
  2. Setting lxd_container_interface_key in gobetween.toml to a value such as user.gobetween.iface and then setting this metadata in each container to a value such as eth1.
• You can use IPv6 by setting lxd_container_address_type in gobetween.toml to IPv6.

[yyyar]

@jtopjian 👍 thanks!