[helm] regression: logs are not being tailed in 1.9.0

[stanislav-zaprudskiy]

Expected behavior

skaffold run --tail tails containers logs

Actual behavior

skaffold run --tail doesn't tail containers logs.

However! If I downgrade to 1.8.0, it tails just fine using the same command line and skaffold.yaml. With debug verbosity, the messages below just don't show up in 1.9.0 after port forwarding has been started:

INFO[0006] Streaming logs from pod: bpid-69d8988d46-k2p9d container: web
INFO[0006] Streaming logs from pod: bpid-69d8988d46-k2p9d container: init
INFO[0006] Streaming logs from pod: bpid-69d8988d46-k2p9d container: app
DEBU[0006] Running command: [kubectl --context docker-desktop logs --since=7s -f bpid-69d8988d46-k2p9d -c init --namespace default]
DEBU[0006] Running command: [kubectl --context docker-desktop logs --since=7s -f bpid-69d8988d46-k2p9d -c web --namespace default]
DEBU[0006] Running command: [kubectl --context docker-desktop logs --since=7s -f bpid-69d8988d46-k2p9d -c app --namespace default]

I also tried skaffold dev --tail without any luck.

Information

• Skaffold version: 1.9.0
• Operating system: macOS Catalina
• Contents of skaffold.yaml:

apiVersion: skaffold/v2alpha4
kind: Config
build:
  tagPolicy:
    sha256: {}
  local:
    concurrency: 0
  artifacts:
    - image: bpid-app
      context: .
      custom:
        buildCommand: |-
          devops/skaffold/docker-build.bash devops/docker/app/Dockerfile devops/skaffold/.env.app
        dependencies:
          dockerfile:
            path: devops/docker/app/Dockerfile
            buildArgs:
              COMPOSER_AUTH: '{"gitlab-token": {"gitlab.com": "{{.GITLAB_PERSONAL_ACCESS_TOKEN}}"}}'
    - image: bpid-web
      context: .
      custom:
        buildCommand: |-
          devops/skaffold/docker-build.bash devops/docker/web/Dockerfile devops/skaffold/.env.web
        dependencies:
          dockerfile:
            path: devops/docker/app/Dockerfile

# https://skaffold.dev/docs/pipeline-stages/port-forwarding/
portForward:
  - resourceType: Service
    resourceName: bpid-mysql
    port: 3306
    localPort: 30336

profiles:
  - name: local-k8s
    activation:
      - kubeContext: microk8s
      - kubeContext: docker-desktop
      - kubeContext: docker-for-desktop
    deploy:
      helm:
        releases:
          - name: bpid
            chartPath: devops/helm-charts/laravel
            wait: true
            skipBuildDependencies: true
            imageStrategy:
              helm: {}
            values:
              image: bpid-app
              nginx.image: bpid-web
            valuesFiles:
              - devops/helm-values/local.yaml
              - devops/helm-values/local-secrets.yaml
              - devops/helm-values/local-personal.yaml
        flags:
          install:
            - --dep-up
            - --atomic
            - --timeout=120
          upgrade:
            - --install
            - --atomic
            - --cleanup-on-fail
            - --force
            - --timeout=60

Steps to reproduce the behavior

Compare output of skaffold run --tail or skaffold run --tail=true in 1.9.0 and 1.8.0.

[dgageot]

I'm the one who broke that... I fixed an issue where logging and port forwarding wouldn't take the runId into account. It was fixed by adding a selector on the label that carries the runID. However, I forgot that our labelling mechanism is flakey with helm. Those labels get applied later and that will lead to lost logs.

Do you still see logs after some time?

[dgageot]

@stanislav-zaprudskiy which version of helm are you using?

[dgageot]

I think it's worst than what I thought. Logging is broken for every type of Kubernetes resource but a simple pod.

[dgageot]

I take that back. The problem is only with Helm. Log tailing pods or deployments deployed with kubectl is ok.

[dgageot]

We have at least three ways to fix that:

• Rollback Only listen to pods for the current RunID #4097 which reintroduces a slightly less important issue
• Evolve labelDeployResults so that it can patch pods' specs too. That should fix the logging but will, I think, spawn new pods which is kind of a glitch.
• For helm3, we might be able to use helm template instead of helm install. That should give us the rendered yaml that we can then patch like we do for kubectl.

[stanislav-zaprudskiy]

Logs didn't show up after 1h. @dgageot

$ helm version
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}

[briandealwis]

We can't use helm template — it doesn't play well with upgrade and delete. But helm 3.1 supports a filtering/transforming the manifest as part of installation (#2350 (comment)).

[dgageot]

Maybe a —post-renderer then?

…

On Wed 6 May 2020 at 15:26 Brian de Alwis ***@***.***> wrote: We can't use helm template — it doesn't play well with upgrade and delete. But helm 3.1 supports a filtering/transforming the manifest as part of installation (#2350 (comment) <#2350 (comment)> ). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4117 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABFPF4WIOTCWQMGDJBZFRDRQFQJJANCNFSM4M2GJMJQ> .

[nkubala]

let's rollback the change that introduced this for now. I would be interested in exploring ways we can interact with the manifests generated by helm though, if we can tease out templated manifests before they get sent off by helm that would be a big win.