CKF 1.7 - VSCode notebook - stuck on creating new notebook and run python code in interactive window

[Barteus]

Bug Description

When creating a new notebook the screen is stuck on loading (tried for 30minutes).

I can run code or python file in terminal but not in interactive mode:

It hands in both cases.

To Reproduce

1. Deploy CKF 1.7
2. Start a VSCode notebook
3. Create a notebook or run a python file in interactive mode.

Environment

$ snap list
Name Version Rev Tracking Publisher Notes
amazon-ssm-agent 3.2.1630.0 7628 latest/stable/… aws✓ classic
core18 20231027 2812 latest/stable canonical✓ base
core20 20231123 2105 latest/stable canonical✓ base
juju 2.9.46 25672 2.9/stable canonical✓ classic
kubectl 1.29.2 3180 latest/stable canonical✓ classic
lxd 5.0.3-51452c3 26881 5.0/stable/… canonical✓ -
microk8s v1.24.17 5872 1.24/stable canonical✓ classic
snapd 2.61.1 20671 latest/stable canonical✓ snapd

Bundle

$ juju export-bundle
bundle: kubernetes
applications:
  admission-webhook:
    charm: admission-webhook
    channel: 1.7/stable
    revision: 224
    series: focal
    resources:
      oci-image: 227
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  argo-controller:
    charm: argo-controller
    channel: 3.3/stable
    revision: 376
    series: focal
    resources:
      oci-image: 326
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  argo-server:
    charm: argo-server
    channel: 3.3/stable
    revision: 309
    series: focal
    resources:
      oci-image: 310
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  dex-auth:
    charm: dex-auth
    channel: 2.31/stable
    revision: 389
    series: focal
    resources:
      oci-image: 457
    scale: 1
    options:
      public-url: http://10.64.140.43.nip.io
      static-password: admin
      static-username: admin
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  istio-ingressgateway:
    charm: istio-gateway
    channel: 1.16/stable
    revision: 663
    series: focal
    scale: 1
    options:
      kind: ingress
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  istio-pilot:
    charm: istio-pilot
    channel: 1.16/stable
    revision: 662
    series: focal
    scale: 1
    options:
      default-gateway: kubeflow-gateway
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  jupyter-controller:
    charm: jupyter-controller
    channel: 1.7/stable
    revision: 805
    series: focal
    resources:
      oci-image: 981
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  jupyter-ui:
    charm: jupyter-ui
    channel: 1.7/stable
    revision: 739
    series: focal
    resources:
      oci-image: 740
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  katib-controller:
    charm: katib-controller
    channel: 0.15/stable
    revision: 282
    series: focal
    resources:
      oci-image: 516
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  katib-db:
    charm: mysql-k8s
    channel: 8.0/stable
    revision: 113
    series: jammy
    resources:
      mysql-image: 103
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64 mem=2048
    storage:
      database: kubernetes,1,1024M
    trust: true
  katib-db-manager:
    charm: katib-db-manager
    channel: 0.15/stable
    revision: 253
    series: focal
    resources:
      oci-image: 257
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  katib-ui:
    charm: katib-ui
    channel: 0.15/stable
    revision: 267
    series: focal
    resources:
      oci-image: 508
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  kfp-api:
    charm: kfp-api
    channel: 2.0-alpha.7/stable
    revision: 935
    series: focal
    resources:
      oci-image: 939
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  kfp-db:
    charm: mysql-k8s
    channel: 8.0/stable
    revision: 113
    series: jammy
    resources:
      mysql-image: 103
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64 mem=2048
    storage:
      database: kubernetes,1,1024M
    trust: true
  kfp-persistence:
    charm: kfp-persistence
    channel: 2.0-alpha.7/stable
    revision: 939
    series: focal
    resources:
      oci-image: 948
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  kfp-profile-controller:
    charm: kfp-profile-controller
    channel: 2.0-alpha.7/stable
    revision: 899
    series: focal
    resources:
      oci-image: 905
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  kfp-schedwf:
    charm: kfp-schedwf
    channel: 2.0-alpha.7/stable
    revision: 952
    series: focal
    resources:
      oci-image: 957
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  kfp-ui:
    charm: kfp-ui
    channel: 2.0-alpha.7/stable
    revision: 934
    series: focal
    resources:
      oci-image: 724
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  kfp-viewer:
    charm: kfp-viewer
    channel: 2.0-alpha.7/stable
    revision: 964
    series: focal
    resources:
      oci-image: 742
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  kfp-viz:
    charm: kfp-viz
    channel: 2.0-alpha.7/stable
    revision: 889
    series: focal
    resources:
      oci-image: 896
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  knative-eventing:
    charm: knative-eventing
    channel: 1.8/stable
    revision: 345
    series: focal
    scale: 1
    options:
      namespace: knative-eventing
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  knative-operator:
    charm: knative-operator
    channel: 1.8/stable
    revision: 320
    series: focal
    resources:
      knative-operator-image: 328
      knative-operator-webhook-image: 198
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  knative-serving:
    charm: knative-serving
    channel: 1.8/stable
    revision: 346
    series: focal
    scale: 1
    options:
      istio.gateway.name: kubeflow-gateway
      istio.gateway.namespace: kubeflow
      namespace: knative-serving
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  kserve-controller:
    charm: kserve-controller
    channel: 0.10/stable
    revision: 458
    series: focal
    resources:
      kserve-controller-image: 441
      kube-rbac-proxy-image: 440
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  kubeflow-dashboard:
    charm: kubeflow-dashboard
    channel: 1.7/stable
    revision: 439
    series: focal
    resources:
      oci-image: 663
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  kubeflow-profiles:
    charm: kubeflow-profiles
    channel: 1.7/stable
    revision: 336
    series: focal
    resources:
      kfam-image: 551
      profile-image: 549
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  kubeflow-roles:
    charm: kubeflow-roles
    channel: 1.7/stable
    revision: 148
    series: focal
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  kubeflow-volumes:
    charm: kubeflow-volumes
    channel: 1.7/stable
    revision: 204
    series: focal
    resources:
      oci-image: 204
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  metacontroller-operator:
    charm: metacontroller-operator
    channel: 2.0/stable
    revision: 204
    series: focal
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  minio:
    charm: minio
    channel: ckf-1.7/stable
    revision: 214
    series: focal
    resources:
      oci-image: 496
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    storage:
      minio-data: kubernetes,1,10240M
  oidc-gatekeeper:
    charm: oidc-gatekeeper
    channel: ckf-1.7/stable
    revision: 320
    series: focal
    resources:
      oci-image: 432
    scale: 1
    options:
      public-url: http://10.64.140.43.nip.io
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  seldon-controller-manager:
    charm: seldon-core
    channel: 1.15/stable
    revision: 548
    series: focal
    resources:
      oci-image: 653
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
  tensorboard-controller:
    charm: tensorboard-controller
    channel: 1.7/stable
    revision: 156
    series: focal
    resources:
      oci-image: 156
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  tensorboards-web-app:
    charm: tensorboards-web-app
    channel: 1.7/stable
    revision: 158
    series: focal
    resources:
      oci-image: 159
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
  training-operator:
    charm: training-operator
    channel: 1.6/stable
    revision: 305
    series: focal
    resources:
      training-operator-image: 322
    scale: 1
    annotations:
      bundleURL: ch:amd64/kubeflow-409
    constraints: arch=amd64
    trust: true
relations:
- - argo-controller:object-storage
  - minio:object-storage
- - dex-auth:oidc-client
  - oidc-gatekeeper:oidc-client
- - istio-pilot:ingress
  - dex-auth:ingress
- - istio-pilot:ingress
  - jupyter-ui:ingress
- - istio-pilot:ingress
  - katib-ui:ingress
- - istio-pilot:ingress
  - kfp-ui:ingress
- - istio-pilot:ingress
  - kubeflow-dashboard:ingress
- - istio-pilot:ingress
  - kubeflow-volumes:ingress
- - istio-pilot:ingress
  - oidc-gatekeeper:ingress
- - istio-pilot:ingress-auth
  - oidc-gatekeeper:ingress-auth
- - istio-pilot:istio-pilot
  - istio-ingressgateway:istio-pilot
- - istio-pilot:ingress
  - tensorboards-web-app:ingress
- - istio-pilot:gateway-info
  - tensorboard-controller:gateway-info
- - istio-pilot:gateway-info
  - kserve-controller:ingress-gateway
- - katib-db-manager:relational-db
  - katib-db:database
- - kfp-api:relational-db
  - kfp-db:database
- - kfp-api:kfp-api
  - kfp-persistence:kfp-api
- - kfp-api:kfp-api
  - kfp-ui:kfp-api
- - kfp-api:kfp-viz
  - kfp-viz:kfp-viz
- - kfp-api:object-storage
  - minio:object-storage
- - kfp-profile-controller:object-storage
  - minio:object-storage
- - kfp-ui:object-storage
  - minio:object-storage
- - kubeflow-profiles:kubeflow-profiles
  - kubeflow-dashboard:kubeflow-profiles

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5459.

This message was autogenerated

[kimwnasptd]

I tried this with upstream Kubeflow 1.7 and managed to run a jupyter server within VSCode (using the base Python 3.8 environment from /usr/bin/python3).

Next step will be to understand if this only happens in CKF. But would be weird since we use the same upstream image, so maybe something with MicroK8s networking could affect this?

[kimwnasptd]

I think I got to the bottom of this and why it happens only on CKF + microK8s and not with upstream KF and KinD.

The reason for all the weird behaviours we see in VSCode, like #340, is because the Service Workers in the browser fail to start in CKF + microK8s compared to KinD and upstream KF.

Specifically, by opening the dev tools in the browser I see the following:

Then with a quick search I found the following issue coder/code-server#4421, which is spot on!

The problem they describe is that:

• The plugins should work if the domain is localhost, which is the case when accessing upstream KF with KinD
• The plugins need HTTPS if they are deployed under a domain

To confirm this we'll need to enable HTTPS and see if the error persists

[Barteus]

Here is the issue we had when trying SSL-enabled CKF (on Charmed K8s + F5 LB integration).

Error loading webview: Error: Could not register service worker: SecurityError: Failed to register a ServiceWorker for scope ('https://kubeflow.xxx.xxx/notebook/admin/ssl-test/stable-2eba7af117ea58d45a6c6449ee4fe63c8d4d53aa/static/out/vs/workbench/contrib/webview/browser/pre/') with script ('https://kubeflow.xxx.xxx/notebook/admin/ssl-test/stable-2eba7af117ea58d45a6c6449ee4fe63c8d4d53aa/static/out/vs/workbench/contrib/webview/browser/pre/service-worker.js?v=4&vscode-resource-base-authority=vscode-resource.vscode-cdn.net&remoteAuthority=kubeflow.xxx.xxx'): An SSL certificate error occurred when fetching the script..

[kimwnasptd]

To be more exact here, this is a problem with Service Workers working either on localhost or with HTTPS, which are used by different plugins.

With my current understanding the solution to this will be for end users to ensure they:

1. Are using HTTPS for VSCode
2. Their browser trusts the certificate, to avoid Service Workers SSL issues

[gustavosr98]

I did not seem to hit this when enabling SSL on Kubeflow and adding the cert to my browser

[kimwnasptd]

Closing this issue since we don't have control over the user's browser and its trust settings