Intermittent python: can't open file '/app/manage.py': [Errno 2] No such file or directory

[scheung38]

Expected behavior

k8s manifest files to be generated

Actual behavior

Intermittently getting:

deployment/django: container arthur-production-django terminated with exit code 2
pod/django-697ddc88b9-hks6r: container arthur-production-django terminated with exit code 2
[django-697ddc88b9-hks6r arthur-production-django] python: can't open file '/app/manage.py': [Errno 2] No such file or directory
deployment/django failed. Error: container arthur-production-django terminated with exit code 2.

• Skaffold version: v1.38.0
• Operating system: Mac OSX 12.4
• Installed via: brew install skaffold
• Contents of skaffold.yaml:

skaffold.yaml:

apiVersion: skaffold/v2beta28
kind: Config
metadata:
  name: skaffold
deploy:
  kubectl:
    manifests:
    - django-claim0-persistentvolumeclaim.yaml
    - django-claim1-persistentvolumeclaim.yaml
    - django-deployment.yaml
    - django-service.yaml
    - envs--production--django-configmap.yaml
    - envs--production--mongodb-configmap.yaml
    - mongo-deployment.yaml
    - mongo-service.yaml
    - production-mongodb-data-persistentvolumeclaim.yaml
    - production-traefik-persistentvolumeclaim.yaml
    - traefik-deployment.yaml
    - traefik-service.yaml

docker-compose-prod.yaml:

version: '3'

volumes:
  production_mongodb_data:
  production_traefik: {}

services:
  django:
    build:
      context: .
      dockerfile: ./compose/production/django/Dockerfile
    image: arthuracr.azurecr.io/arthur_production_django:v1.0.0
    
    container_name: arthur_production_django
    # platform: linux/x86_64
    depends_on:
      # - postgres
      # - redis
      - mongo
    volumes:
      - .:/app:z
      - ../RTL_data/_PAF_Data_samples/PAF_MAIN_FILE:/uploaddata

    env_file:
      - ./.envs/.production/.django
      # - ./.envs/.production/.postgres
      - ./.envs/.production/.mongodb
    ports:
      - "8000:8000"
      - "3000:3000"
    command: /start
    stdin_open: true
    tty: true

  mongo:
    image: arthuracr.azurecr.io/mongo:5.0.6
    container_name: "mongo"
    restart: always
    env_file:
      - ./.envs/.production/.mongodb
    environment:
      - MONGO_INITDB_ROOT_USERNAME=XXXX
      - MONGO_INITDB_ROOT_PASSWORD=XXXX
      - MONGO_INITDB_DATABASE= XXXX 
      - MONGO_INITDB_USERNAME=XXXX 
      - MONGO_INITDB_PASSWORD=XXXX 
    volumes:
      - production_mongodb_data:/data/db
      # - ${PWD}/_data/mongo:/data/db
      # - ${PWD}/docker/_mongo/fixtures:/import
      # - ${PWD}/docker/_mongo/scripts/init.sh:/docker-entrypoint-initdb.d/setup.sh
    ports:
      - 27017:27017

  traefik:
    build:
      context: .
      dockerfile: ./compose/production/traefik/Dockerfile
    image: arthuracr.azurecr.io/arthur_production_traefik:v2.2.11
    container_name: arthur_production_traefik
    depends_on:
      - django
    volumes:
      - production_traefik:/etc/traefik/acme:z
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"
    # command: --api.insecure=true --providers.docker

Steps to reproduce the behavior

1. a clonable repository with the sample skaffold project
2. skaffold dev --default-repo $MY_ACR
3. ...

Listing files to watch...
Generating tags...
Checking cache...
Tags used in deployment:
Starting deploy...
 - persistentvolumeclaim/django-claim0 created
 - persistentvolumeclaim/django-claim1 created
 - deployment.apps/django created
 - service/django created
 - configmap/envs--production--django created
 - configmap/envs--production--mongodb created
 - deployment.apps/mongo created
 - service/mongo created
 - persistentvolumeclaim/production-mongodb-data created
 - persistentvolumeclaim/production-traefik created
 - deployment.apps/traefik created
 - service/traefik created
Waiting for deployments to stabilize...
 - deployment/django: creating container arthur-production-django
    - pod/django-697ddc88b9-hks6r: creating container arthur-production-django
 - deployment/mongo: creating container mongo
    - pod/mongo-7775588866-n6zqn: creating container mongo
 - deployment/traefik: creating container arthur-production-traefik
    - pod/traefik-9668b69fc-s9cpb: creating container arthur-production-traefik
 - deployment/traefik is ready. [2/3 deployment(s) still pending]
 - deployment/django: container arthur-production-django terminated with exit code 2
    - pod/django-697ddc88b9-hks6r: container arthur-production-django terminated with exit code 2
      > [django-697ddc88b9-hks6r arthur-production-django] python: can't open file '/app/manage.py': **[Errno 2] No such file or directory**
 - deployment/django failed. Error: container arthur-production-django terminated with exit code 2.
Cleaning up...
 - persistentvolumeclaim "django-claim0" deleted
 - persistentvolumeclaim "django-claim1" deleted
 - deployment.apps "django" deleted
 - service "django" deleted
 - configmap "envs--production--django" deleted
 - configmap "envs--production--mongodb" deleted
 - deployment.apps "mongo" deleted
 - service "mongo" deleted
 - persistentvolumeclaim "production-mongodb-data" deleted
 - persistentvolumeclaim "production-traefik" deleted
 - deployment.apps "traefik" deleted
 - service "traefik" deleted
1/3 deployment(s) failed

Is this related to periodSeconds or timeoutSeconds to specified? That django is waiting for MongoDB when it was not ready etc? livelinessProbe or readinessProbe is the solution ?

[aaron-prindle]

@scheung38 were you able to resolve this issue? From the logs here it is difficult to tell what might be happening. Can you post skaffold logs with verbose logging as well:

skaffold dev -v debug --default-repo $MY_ACR

Additionally do you have a small sharable repo that reproduces this? Thanks

[ericzzzzzzz]

hi @scheung38 were you able to resolve the issue? Could you try the latest skaffold to see if the issue still occurs?