Questions & Suggestions regarding Cloud Code

[tvvignesh]

Hi. I have been using Skaffold for quite a while now and Cloud code's support for Skaffold with VSCode looks awesome. I did try installing and exploring the features. Looks great, I just had a few questions for which I was looking for answers. Also had some suggestions/ideas. Sharing it all here:

1. I have been using Azure Kubernetes Tools extension so far to work with K8 clusters in both my local Kind cluster as well my remote GKE cluster. Is it possible to provide some comparison on how both of these differ (pros/cons) - would be useful info for users trying to choose either. One pro I definitely see of Azure kubernetes tools is the licensing (which is open source MIT) whereas this is proprietary and the pro I see here is the great integration with Skaffold. Is there any comparison available?

2. I have been also closely following Mindaro a.k.a Bridge for Kubernetes (https://github.com/microsoft/mindaro) , again from Microsoft which is in early development. Does cloud code support something like that? Have everything in remote cluster and take just what you are working on and work on it locally? (Hybrid development). This would greatly help since as the application grows, especially in a microservices environment, it is not practical to have all services running locally. And complete remote development would also be an issue. So, something like this would help since it avoids the mess of having to provision separate namespaces for every developer.

3. Since Cloud Code already has integration with GCP, an integration with GCE would also be great here. I have my complete remote dev environment hosted in GCE and I access everything else from there. I use pre-emptible GCE VMs with boot disk and data disk attached to the VM (so my dev env is disposable). I currently do operations like starting VM, stopping VM via Powershell scripts but it would be great to have Cloud code support that as well.

4. Also, in the same context, I use IAP tunneling for everything. So, I run a Powershell script again to start my IAP tunnel after which I do a VSCode remote SSH via the tunnel onto my development VM from which I do everything. Maybe, Cloud code can provide me a config to automatically tunnel when I use remote SSH.

5. Also, whenever I start my cluster, I have to deploy multiple services be it remote/local. Currently, I am doing this using Tilt locally and I am not sure what's the best way to do this remotely, but Cloud Code or Skaffold providing this feature would be great. I see that this has landed in Icebox milestone with Skaffold here: Better handling for multiple microservices (ability to handle multiple skaffold.yaml files) GoogleContainerTools/skaffold#167 and another issue here: Support parent/child "module" spec inlined in a single skaffold.yaml GoogleContainerTools/skaffold#4457 - this is important to manage a microservices architecture.

6. In the same context, it would be great if Cloud Code handles this: https://github.com/GoogleContainerTools/skaffold/blob/master/ROADMAP.md#interactive-ui---p3 so developers can avoid a separate UI or context switching but get a context of all services running in the cluster without having to manually browse open the logs of every service.

7. Also, regarding Port Forwarding, while I see this issue: FR: Allow port-forward and attaching terminal to pods/services #47 where Skaffold and Run on Kubernetes, the problem is that the Port forwarding rules need to be predefined. While I can definitely specify the port forwarding rules for significant pods before hand, not all ports would typically be specified. Having the ability to dynamically specify ports to forward would help rather than adding it to the config. This is how Remote SSH and Azure VSCode Kubernetes tools does forwarding as well. Right click -> Forward Port -> Specify port mapping -> Done

Thanks in advance. And sorry for the long thread 😅

[sivakku]

Thanks Vignesh(@tvvignesh) for detailed and thoughtful suggestions and questions:

(1) Deeper integrations with Minikube (local development, auto state transitions and cleanups), simplification of GCP auth, secrets mgmt, testing them locally with out deploying, GCP API usage right out of IDE, integration with serverless compute (like cloud run) with a similar flow as K8. The IDE plugins are free, we are re-evaluating about open sourcing for more community engagement.

(2) We have been thinking about something similar for testing in conjunction with micro services development where all of them can’t fit in the box. Today they split out a public URL service endpoints for every run which means you can launch in a remote environment and have other environments to talk to this one using public URL/interfaces. We will at some time simplify the upstream routing that does the routing magic in a subset of micro-services but downstream routing should be already there.

(3) What is the scenario you are trying to use the remote VM for? Are you planning to use the remote GCE VM as a dev environment for IDE(VSCode) (or) Are you trying to work with multiple GCE VMs and from IDE(VSCode) you need a way to browse and perform operations on each VM across pool of VMs? For the first one did you try our remote shell development environment that plugins work with(https://cloud.google.com/code/docs/vscode/quickstart-remote-dev).

(4) I think the above tutorial is what you are looking for, in the remote shell if you open any terminal that one is connected to your remote box and you can do all your development there. But do let me know if that is not what you are looking for.

(5) I haven’t played with tilt for orchestration of services, the skaffold configuration is planning to take multiple set of environment and they map to set of services that can be deployed sometime in their roadmap. Today in the project you can have multiple services and we allow deploying to all of them at once. I am going to follow up on this one and get back.

(6) This multiple different phases/diagnostics is going to be coming in to cloud code in a native VSCode UX, some time in early next year. What problem do you currently see when trying to deploy using IDE plugins since those stage are broken down in a different way in the output window?

(7) Going to create a separate task for the dynamic port forwarding feature.

[tvvignesh]

@sivakku Thanks for your quick amd elaborate reply.

1. Open sourcing would be great for people like me to have more confidence in the development process - and also contribute sometime in the future. Great to see that you are rethinking about it. Maybe if you have proprietary code that you don't want to share, you can selectively add them as more extensions integrating to Cloud Code. I use Kind for development (not sure about the extent of support for that though I am able to see and do all operations in my cluster). So overall I think the differentiating factor is the deeper integration with GCP if I am not wrong.

2. Sounds great. Ultimately my final aim is to just pick the service I am working on from the remote cluster, develop locally and then finally deploy it without worrying about any complexity behind the scenes. I have tried/researched many tools in this area to see which can work well for me but still it feels like I have not found the right fit -

a) Telepresence - Does not cater to multiple developers on the same namespace

b) Service Preview - Addresses this but since I use Restricted PSP in my cluster, I was not able to use it. Also it is not open source, with free subscription upto 5 devs,

c) Mindaro - Does exactly this but facing quite a few challenges since it is in its early stages - You can see Issue No. 63 which I filed in their repo and they are actively working on it. But the great thing here is that, it integrates with Azure Kubernetes Tools extension hence making it a platform for other extensions to work on.

d) Tilt - Good for local development but I still do face some challenges which I am discussing with the Tilt team. But does not provide hybrid devex like Mindaro with no integration currently with VSCode having me to switch contexts regularly. Also, since it currently does not cater prod deployments, I have to maintain both a Tiltfile and Skaffold file.

e) Okteto - While they are doing great work (and I have not tried it personally), the problem I see here is the need to have separate namespaces for every developer.

f) Skaffold - Great for local dev and a bit challenging for remote dev and does not currently do hybrid dev

So, as of now, I am just hoping I can have one tool (preferrably with VSCode) to do all the job for me.

3. I have my remote VM running Ubuntu where I remote ssh into using VSCode and all the development I do is within that. My local machine just acts as a way to access the VM. While I did look at https://cloud.google.com/code/docs/vscode/quickstart-remote-dev the reason it solely does not cater to my needs is cause its not just kubernetes which is part of my dev workflow. I use a lot of tools around Kubernetes. So, this remote VM becomes my remote dev environment (think of it like a remote Vagrant box where I have everything setup). So, even if I switch my local machines, all I have to do is authenticate with GCloud client and install VSCode locally (everything else is in remote). I use a Windows host, Ubuntu 20.04 remote for development

4. As mentioned above, my workflow is quite different. And while I will definitely use K8 remote development from cloud code, I was looking for easier access to my complete dev environment

5. While all of them can be deployed through something like shell scripts running skaffold dev in every directory , there are some problems like no guarantee of orders, and more as documented here: https://tinyurl.com/skaffold-modules Anyways, I guess they are already working on this

6. Nice to hear that. Tilt does a good job here doing things like alerting me when any of the services I deployed throws errors which need not lead to pod crash necessarily. So, currently if I am working on a service, I will have to manually to open the logs of another related service or run the debugger on it to understand that there was an error or have logs of every service open in multiple tabs. While this can work for small number of microservices, things will start falling apart when there are lots of them which is why people opt in for distributed tracing or a logging stack. But if I can do something similar during dev, that would be easy (Not sure about this, but maybeI should use jaeger for dev as well).

Hope this clarifies. Thanks.

[tvvignesh]

Btw 3 and 4 are not critical for me since the scripts already work well for me. The rest are important to have.

[tvvignesh]

@sivakku Hi again. I just tried cloud code properly again and I am facing some issues.

Cloud code deployed everything properly onto the cluster.

1. If I start debugging, port forwarding starts, but if I stop debugging, port forwarding stops - I try to close all the VSCode windows containing all microservices that I am not currently developing on and have only the relevant windows open (I don't use one repo for all microservices) - But what I want to do is still be able to forward the ports of those microservices which doesn't happen since forwarding only happens when the window is open and debugging is started which is a bit inconvenient. I have to end up forwarding some other way manually without being able to leverage what skaffold does for me already or open up multiple VSCode windows for this.

2. I have disabled cleanup since I was confused what application termination means here. Does it mean deleting the deployment or exiting VSCode or debugger? Cause if I understand correctly, Cloud code is running skaffold dev , skaffold debug and all related commands for me. So I am not sure when cloud code stops skaffold dev behind the scenes.

In the same context, a way to manually trigger cleanup would be good to have.

Apart from this, it was really great. I will share more feedback as I try it.

[tvvignesh]

Another issue I see is that, even if I install the latest version of skaffold, for eg. 1.17.1 I see the terminal using an older version of skaffold as shipped by Cloud Code eg. 1.16.0.

For instance /usr/local/bin/skaffold is what I installed, and /home/vignesh/.cache/cloud-code/installer/google-cloud-sdk/bin/skaffold is what cloud code comes with I guess (Don't find it added in both my ~/.bashrc or ~/.profile files)

When I do whereis skaffold I get this:

skaffold: /usr/local/bin/skaffold /home/vignesh/.cache/cloud-code/installer/google-cloud-sdk/bin/skaffold

Though I understand that Cloud code may depend on a specific version of skaffold, I am wondering why its overriding the global path variables.

[tvvignesh]

Sorry, making this thread huge. Just wanted to consolidate all my suggestions here as I work with Cloud code.

When you go to logs viewer (through the picker) rather than drilling through the explorer, selecting the right pod and clicking on View Logs the filters are empty. Rather the filters can be inferred from the skaffold file being used.

This will allow me to do things like add a keybinding to Logs Viewer, then I can just get into the project I want in VSCode and press the Keybinding to get the logs. A minor but significant usability improvement.

[tvvignesh]

GKE offers me checkboxes to select multiple resources and do bulk operations (like bulk delete). The Cloud Code Cluster explorer can support multiple operations like these as well rather than having to do it via the UI (this is not a priority though, but good to have).

[tvvignesh]

When the Log viewer shows an error like the one below, the Additional filters do not have any data to select from even when the cluster is active and other pods and deployments are running.

But it shows up when I don't get an error like this:

[tvvignesh]

The Log Viewer only has options to filter pods within a deployment, but not statefulsets. You also don't get the Context menu when you right click in the cluster explorer.

Also, if I am not wrong, none of the logs are shown for pods from Stateful sets, even if you go to the pod level and go to View Logs. Tried the same with different statefulsets and have the same result. It works for deployments though.

[tvvignesh]

I find Cloud Code Explorer to be very slow (not sure why). So, when I went to the console to see if there were any logs, I find these huge logs printed repeatedly that the console itself hangs when scrolling through them.

Also the sourcemap does not exist in the path it is looking for, I get these logs in the console: