frigate.watchdog detection appears to have stopped #10764

oldhome7 · 2024-03-31T21:47:20Z

oldhome7
Mar 31, 2024

Describe the problem you are having

I'm running the tensorrt image with a Quadro K620 for the detector and ffmpeg and about every minute I get a dropped connection and I've looked through the container log and noticed this seems to be the issue.

If I switch to the regular image and comment out the tensor detector and just use cpu it runs fine.

Version

0.13.2-6476f8a

Frigate config file

mqtt:
  enabled: False

detectors:
#  cpu:
#    type: cpu
#    num_threads: 6
#    model:
#      path: "/custom_model.tflite"

  tensorrt:
    type: tensorrt
    device: 0 #This is the default, select the first GPU

model:
  path: /trt-models/yolov7-tiny-416.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 416
  height: 416

ffmpeg:
  hwaccel_args: preset-nvidia-h264

cameras:
  camera1: # <--- this will be changed to your actual camera later
    enabled: True
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED:REDACTED@192.168.1.172:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
            - record
      output_args:
        record: preset-record-generic-audio-copy
    detect:
      enabled: false # <---- disable detection until you have a working camera feed
      width: 704 # <---- update for your camera's resolution
      height: 480 # <---- update for your camera's resolution

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
    shm_size: "16384mb" # update for your cameras based on calculation above
#    devices:
#      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
#      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-o>
#      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config:rw
      - /mnt/MEDIA/Frigate:/media/frigate
      - ./database:/db
      - ./trt-models:/trt-models
#      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
#        target: /tmp/cache
#        tmpfs:
#          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "REDACTED"

    deploy:
      resources:
        reservations:
          devices:
          - driver: "nvidia"
            device_ids: ['0']
            capabilities: [gpu]

Relevant log output

{"log":"2024-03-31 16:42:57.795934348  [2024-03-31 16:42:57] frigate.app                    INFO    : Starting Frigate (0.13.2-6476f8a)\n","stream":"stdout","time":"2024-03-31T21:42:57.796468737Z"}
{"log":"2024-03-31 16:42:58.127820005  [2024-03-31 16:42:58] peewee_migrate.logs            INFO    : Starting migrations\n","stream":"stdout","time":"2024-03-31T21:42:58.128377117Z"}
{"log":"2024-03-31 16:42:58.147119795  [2024-03-31 16:42:58] peewee_migrate.logs            INFO    : There is nothing to migrate\n","stream":"stdout","time":"2024-03-31T21:42:58.147547389Z"}
{"log":"2024-03-31 16:42:58.164209102  [2024-03-31 16:42:58] frigate.app                    INFO    : Recording process started: 450\n","stream":"stdout","time":"2024-03-31T21:42:58.164585222Z"}
{"log":"2024-03-31 16:42:58.175989065  [2024-03-31 16:42:58] frigate.app                    INFO    : go2rtc process pid: 98\n","stream":"stdout","time":"2024-03-31T21:42:58.176346068Z"}
{"log":"2024-03-31 16:42:58.246760527  [2024-03-31 16:42:58] frigate.app                    INFO    : Output process started: 460\n","stream":"stdout","time":"2024-03-31T21:42:58.247256227Z"}
{"log":"2024-03-31 16:42:58.279553818  [2024-03-31 16:42:58] detector.tensorrt              INFO    : Starting detection process: 459\n","stream":"stdout","time":"2024-03-31T21:42:58.279955911Z"}
{"log":"2024-03-31 16:42:58.284817266  [2024-03-31 16:42:58] frigate.app                    INFO    : Camera processor started for camera1: 467\n","stream":"stdout","time":"2024-03-31T21:42:58.285215284Z"}
{"log":"2024-03-31 16:42:58.300512939  [2024-03-31 16:42:58] frigate.app                    INFO    : Capture process started for camera1: 469\n","stream":"stdout","time":"2024-03-31T21:42:58.300854545Z"}
{"log":"2024-03-31 16:42:58.555105298  [2024-03-31 16:42:58] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 36 MiB\n","stream":"stdout","time":"2024-03-31T21:42:58.555611345Z"}
{"log":"2024-03-31 16:42:58.991647895  [2024-03-31 16:42:58] frigate.detectors.plugins.tensorrt ERROR   : 1: [stdArchiveReader.cpp::StdArchiveReader::37] Error Code 1: Serialization (Serialization assertion safeVersionRead == safeSerializationVersion failed.Version tag does not match. Note: Current Version: 0, Serialized Engine Version: 43)\n","stream":"stdout","time":"2024-03-31T21:42:58.99213274Z"}
{"log":"2024-03-31 16:42:59.023686257  Process detector:tensorrt:\n","stream":"stdout","time":"2024-03-31T21:42:59.024129786Z"}
{"log":"2024-03-31 16:42:59.023699275  [2024-03-31 16:42:59] frigate.detectors.plugins.tensorrt ERROR   : 4: [runtime.cpp::deserializeCudaEngine::66] Error Code 4: Internal Error (Engine deserialization failed.)\n","stream":"stdout","time":"2024-03-31T21:42:59.024200792Z"}
{"log":"2024-03-31 16:42:59.026495019  Traceback (most recent call last):\n","stream":"stdout","time":"2024-03-31T21:42:59.027018857Z"}
{"log":"2024-03-31 16:42:59.026503480    File \"/usr/lib/python3.9/multiprocessing/process.py\", line 315, in _bootstrap\n","stream":"stdout","time":"2024-03-31T21:42:59.027108758Z"}
{"log":"2024-03-31 16:42:59.026507075      self.run()\n","stream":"stdout","time":"2024-03-31T21:42:59.027126067Z"}
{"log":"2024-03-31 16:42:59.026511091    File \"/usr/lib/python3.9/multiprocessing/process.py\", line 108, in run\n","stream":"stdout","time":"2024-03-31T21:42:59.027139659Z"}
{"log":"2024-03-31 16:42:59.026514757      self._target(*self._args, **self._kwargs)\n","stream":"stdout","time":"2024-03-31T21:42:59.027153549Z"}
{"log":"2024-03-31 16:42:59.026585632    File \"/opt/frigate/frigate/object_detection.py\", line 102, in run_detector\n","stream":"stdout","time":"2024-03-31T21:42:59.027232869Z"}
{"log":"2024-03-31 16:42:59.026590277      object_detector = LocalObjectDetector(detector_config=detector_config)\n","stream":"stdout","time":"2024-03-31T21:42:59.027265387Z"}
{"log":"2024-03-31 16:42:59.026594033    File \"/opt/frigate/frigate/object_detection.py\", line 53, in __init__\n","stream":"stdout","time":"2024-03-31T21:42:59.027288594Z"}
{"log":"2024-03-31 16:42:59.026597521      self.detect_api = create_detector(detector_config)\n","stream":"stdout","time":"2024-03-31T21:42:59.027313935Z"}
{"log":"2024-03-31 16:42:59.026601374    File \"/opt/frigate/frigate/detectors/__init__.py\", line 18, in create_detector\n","stream":"stdout","time":"2024-03-31T21:42:59.027337856Z"}
{"log":"2024-03-31 16:42:59.026611054      return api(detector_config)\n","stream":"stdout","time":"2024-03-31T21:42:59.027364246Z"}
{"log":"2024-03-31 16:42:59.026614789    File \"/opt/frigate/frigate/detectors/plugins/tensorrt.py\", line 221, in __init__\n","stream":"stdout","time":"2024-03-31T21:42:59.027388733Z"}
{"log":"2024-03-31 16:42:59.026689438      self.input_shape = self._get_input_shape()\n","stream":"stdout","time":"2024-03-31T21:42:59.027413639Z"}
{"log":"2024-03-31 16:42:59.026693955    File \"/opt/frigate/frigate/detectors/plugins/tensorrt.py\", line 93, in _get_input_shape\n","stream":"stdout","time":"2024-03-31T21:42:59.027439196Z"}
{"log":"2024-03-31 16:42:59.026760590      binding = self.engine[0]\n","stream":"stdout","time":"2024-03-31T21:42:59.027459661Z"}
{"log":"2024-03-31 16:42:59.026764800  TypeError: 'NoneType' object is not subscriptable\n","stream":"stdout","time":"2024-03-31T21:42:59.027473013Z"}
{"log":"2024-03-31 16:42:59.026798632  Exception ignored in: \u003cfunction TensorRtDetector.__del__ at 0x7f9c6cef4ca0\u003e\n","stream":"stdout","time":"2024-03-31T21:42:59.027486085Z"}
{"log":"2024-03-31 16:42:59.026802221  Traceback (most recent call last):\n","stream":"stdout","time":"2024-03-31T21:42:59.027500514Z"}
{"log":"2024-03-31 16:42:59.026805886    File \"/opt/frigate/frigate/detectors/plugins/tensorrt.py\", line 239, in __del__\n","stream":"stdout","time":"2024-03-31T21:42:59.027513291Z"}
{"log":"2024-03-31 16:42:59.027281061      if self.outputs is not None:\n","stream":"stdout","time":"2024-03-31T21:42:59.027623131Z"}
{"log":"2024-03-31 16:42:59.027289568  AttributeError: 'TensorRtDetector' object has no attribute 'outputs'\n","stream":"stdout","time":"2024-03-31T21:42:59.027674084Z"}
{"log":"2024-03-31 16:43:03.301333719  [INFO] Starting go2rtc healthcheck service...\n","stream":"stdout","time":"2024-03-31T21:43:03.301864466Z"}
{"log":"2024-03-31 16:43:18.680410703  [2024-03-31 16:43:18] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting Frigate...\n","stream":"stdout","time":"2024-03-31T21:43:18.680968789Z"}
{"log":"s6-rc: info: service legacy-services: stopping\n","stream":"stderr","time":"2024-03-31T21:43:18.688707026Z"}
{"log":"s6-rc: info: service legacy-services successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:18.696073783Z"}
{"log":"s6-rc: info: service nginx: stopping\n","stream":"stderr","time":"2024-03-31T21:43:18.696537549Z"}
{"log":"s6-rc: info: service go2rtc-healthcheck: stopping\n","stream":"stderr","time":"2024-03-31T21:43:18.696901972Z"}
{"log":"2024-03-31 16:43:18.717779254  [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15)\n","stream":"stdout","time":"2024-03-31T21:43:18.718152262Z"}
{"log":"s6-rc: info: service go2rtc-healthcheck successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:18.72017366Z"}
{"log":"2024-03-31 16:43:18.782682328  [INFO] Service NGINX exited with code 0 (by signal 0)\n","stream":"stdout","time":"2024-03-31T21:43:18.783170562Z"}
{"log":"s6-rc: info: service nginx successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:18.785259366Z"}
{"log":"s6-rc: info: service nginx-log: stopping\n","stream":"stderr","time":"2024-03-31T21:43:18.785639975Z"}
{"log":"s6-rc: info: service frigate: stopping\n","stream":"stderr","time":"2024-03-31T21:43:18.78606179Z"}
{"log":"s6-rc: info: service nginx-log successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:18.791320659Z"}
{"log":"2024-03-31 16:43:18.790905569  [2024-03-31 16:43:18] frigate.app                    INFO    : Stopping...\n","stream":"stdout","time":"2024-03-31T21:43:18.79134563Z"}
{"log":"2024-03-31 16:43:18.792988283  [2024-03-31 16:43:18] frigate.stats                  INFO    : Exiting stats emitter...\n","stream":"stdout","time":"2024-03-31T21:43:18.793420164Z"}
{"log":"2024-03-31 16:43:18.793398700  [2024-03-31 16:43:18] frigate.storage                INFO    : Exiting storage maintainer...\n","stream":"stdout","time":"2024-03-31T21:43:18.79373487Z"}
{"log":"2024-03-31 16:43:18.794296696  [2024-03-31 16:43:18] frigate.events.cleanup         INFO    : Exiting event cleanup...\n","stream":"stdout","time":"2024-03-31T21:43:18.794576101Z"}
{"log":"2024-03-31 16:43:18.795112904  [2024-03-31 16:43:18] frigate.record.cleanup         INFO    : Exiting recording cleanup...\n","stream":"stdout","time":"2024-03-31T21:43:18.795497313Z"}
{"log":"2024-03-31 16:43:18.796789556  [2024-03-31 16:43:18] frigate.ptz.autotrack          INFO    : Exiting autotracker...\n","stream":"stdout","time":"2024-03-31T21:43:18.797093901Z"}
{"log":"2024-03-31 16:43:18.797137239  [2024-03-31 16:43:18] frigate.watchdog               INFO    : Exiting watchdog...\n","stream":"stdout","time":"2024-03-31T21:43:18.797391262Z"}
{"log":"2024-03-31 16:43:18.892573478  [2024-03-31 16:43:18] frigate.object_processing      INFO    : Exiting object processor...\n","stream":"stdout","time":"2024-03-31T21:43:18.893034867Z"}
{"log":"2024-03-31 16:43:19.242242551  [2024-03-31 16:43:19] frigate.comms.ws               INFO    : Exiting websocket client...\n","stream":"stdout","time":"2024-03-31T21:43:19.242759315Z"}
{"log":"2024-03-31 16:43:19.696966705  [2024-03-31 16:43:19] frigate.events.maintainer      INFO    : Exiting event processor...\n","stream":"stdout","time":"2024-03-31T21:43:19.697552259Z"}
{"log":"2024-03-31 16:43:20.204267307  [2024-03-31 16:43:20] peewee.sqliteq                 INFO    : writer received shutdown request, exiting.\n","stream":"stdout","time":"2024-03-31T21:43:20.204720584Z"}
{"log":"2024-03-31 16:43:20.211464125  [2024-03-31 16:43:20] frigate.record.maintainer      INFO    : Exiting recording maintenance...\n","stream":"stdout","time":"2024-03-31T21:43:20.211773012Z"}
{"log":"2024-03-31 16:43:20.211883890  [2024-03-31 16:43:20] watchdog.camera1               INFO    : Terminating the existing ffmpeg process...\n","stream":"stdout","time":"2024-03-31T21:43:20.212186955Z"}
{"log":"2024-03-31 16:43:20.212318699  [2024-03-31 16:43:20] watchdog.camera1               INFO    : Waiting for ffmpeg to exit gracefully...\n","stream":"stdout","time":"2024-03-31T21:43:20.212540241Z"}
{"log":"2024-03-31 16:43:20.295881068  [2024-03-31 16:43:20] frigate.video                  INFO    : camera1: exiting subprocess\n","stream":"stdout","time":"2024-03-31T21:43:20.296258173Z"}
{"log":"2024-03-31 16:43:21.939344755  /usr/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 5 leaked shared_memory objects to clean up at shutdown\n","stream":"stdout","time":"2024-03-31T21:43:21.940036619Z"}
{"log":"2024-03-31 16:43:21.939357339    warnings.warn('resource_tracker: There appear to be %d '\n","stream":"stdout","time":"2024-03-31T21:43:21.940126219Z"}
{"log":"2024-03-31 16:43:21.948484825  [INFO] Service Frigate exited with code 0 (by signal 0)\n","stream":"stdout","time":"2024-03-31T21:43:21.948930757Z"}
{"log":"s6-rc: info: service frigate successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:21.952202327Z"}
{"log":"s6-rc: info: service trt-model-prepare: stopping\n","stream":"stderr","time":"2024-03-31T21:43:21.952840948Z"}
{"log":"s6-rc: info: service go2rtc: stopping\n","stream":"stderr","time":"2024-03-31T21:43:21.953237936Z"}
{"log":"s6-rc: info: service frigate-log: stopping\n","stream":"stderr","time":"2024-03-31T21:43:21.953807406Z"}
{"log":"2024-03-31 16:43:21.956060737  exit with signal: terminated\n","stream":"stdout","time":"2024-03-31T21:43:21.956266722Z"}
{"log":"s6-rc: info: service trt-model-prepare successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:21.956277167Z"}
{"log":"s6-rc: info: service frigate-log successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:21.959518794Z"}
{"log":"2024-03-31 16:43:21.969015805  [INFO] The go2rtc service exited with code 0 (by signal 0)\n","stream":"stdout","time":"2024-03-31T21:43:21.969311843Z"}
{"log":"s6-rc: info: service go2rtc successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:21.971167512Z"}
{"log":"s6-rc: info: service go2rtc-log: stopping\n","stream":"stderr","time":"2024-03-31T21:43:21.971740575Z"}
{"log":"s6-rc: info: service go2rtc-log successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:21.977312075Z"}
{"log":"s6-rc: info: service log-prepare: stopping\n","stream":"stderr","time":"2024-03-31T21:43:21.977689763Z"}
{"log":"s6-rc: info: service s6rc-fdholder: stopping\n","stream":"stderr","time":"2024-03-31T21:43:21.978311198Z"}
{"log":"s6-rc: info: service log-prepare successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:21.981749973Z"}
{"log":"s6-rc: info: service legacy-cont-init: stopping\n","stream":"stderr","time":"2024-03-31T21:43:21.982053308Z"}
{"log":"s6-rc: info: service s6rc-fdholder successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:21.983936401Z"}
{"log":"s6-rc: info: service legacy-cont-init successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:21.998051105Z"}
{"log":"s6-rc: info: service fix-attrs: stopping\n","stream":"stderr","time":"2024-03-31T21:43:21.998537503Z"}
{"log":"s6-rc: info: service fix-attrs successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:22.002118491Z"}
{"log":"s6-rc: info: service s6rc-oneshot-runner: stopping\n","stream":"stderr","time":"2024-03-31T21:43:22.00247336Z"}
{"log":"s6-rc: info: service s6rc-oneshot-runner successfully stopped\n","stream":"stderr","time":"2024-03-31T21:43:22.009038112Z"}
{"log":"s6-rc: info: service s6rc-fdholder: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.488112363Z"}
{"log":"s6-rc: info: service s6rc-oneshot-runner: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.488387974Z"}
{"log":"s6-rc: info: service s6rc-oneshot-runner successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.496526761Z"}
{"log":"s6-rc: info: service fix-attrs: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.49692645Z"}
{"log":"s6-rc: info: service s6rc-fdholder successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.499045785Z"}
{"log":"s6-rc: info: service fix-attrs successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.51004118Z"}
{"log":"s6-rc: info: service legacy-cont-init: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.51040851Z"}
{"log":"s6-rc: info: service legacy-cont-init successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.522705164Z"}
{"log":"s6-rc: info: service trt-model-prepare: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.522857146Z"}
{"log":"s6-rc: info: service log-prepare: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.523546947Z"}
{"log":"No models to convert.\n","stream":"stdout","time":"2024-03-31T21:43:28.543340241Z"}
{"log":"s6-rc: info: service trt-model-prepare successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.544452165Z"}
{"log":"s6-rc: info: service log-prepare successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.544852438Z"}
{"log":"s6-rc: info: service nginx-log: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.545383484Z"}
{"log":"s6-rc: info: service go2rtc-log: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.545886517Z"}
{"log":"s6-rc: info: service frigate-log: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.546388968Z"}
{"log":"s6-rc: info: service nginx-log successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.550208238Z"}
{"log":"s6-rc: info: service go2rtc-log successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.550825922Z"}
{"log":"s6-rc: info: service go2rtc: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.551183124Z"}
{"log":"s6-rc: info: service frigate-log successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.551651412Z"}
{"log":"s6-rc: info: service go2rtc successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.556245661Z"}
{"log":"s6-rc: info: service go2rtc-healthcheck: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.556818138Z"}
{"log":"s6-rc: info: service frigate: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.557410587Z"}
{"log":"s6-rc: info: service frigate successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.561718404Z"}
{"log":"s6-rc: info: service nginx: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.562315424Z"}
{"log":"s6-rc: info: service go2rtc-healthcheck successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.562763319Z"}
{"log":"s6-rc: info: service nginx successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.568656712Z"}
{"log":"s6-rc: info: service legacy-services: starting\n","stream":"stderr","time":"2024-03-31T21:43:28.569269808Z"}
{"log":"2024-03-31 16:43:28.575829462  [INFO] Preparing Frigate...\n","stream":"stdout","time":"2024-03-31T21:43:28.576385626Z"}
{"log":"2024-03-31 16:43:28.578323177  [INFO] Starting NGINX...\n","stream":"stdout","time":"2024-03-31T21:43:28.57876138Z"}
{"log":"2024-03-31 16:43:28.582893889  [INFO] Preparing new go2rtc config...\n","stream":"stdout","time":"2024-03-31T21:43:28.583241425Z"}
{"log":"s6-rc: info: service legacy-services successfully started\n","stream":"stderr","time":"2024-03-31T21:43:28.586840766Z"}
{"log":"2024-03-31 16:43:28.610811022  [INFO] Starting Frigate...\n","stream":"stdout","time":"2024-03-31T21:43:28.611290535Z"}
{"log":"2024-03-31 16:43:28.766010726  2024/03/31 16:43:28 [error] 155#155: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.235, server: , request: \"GET /ws HTTP/1.1\", upstream: \"http://127.0.0.1:5002/\", host: \"192.168.1.178:5000\"\n","stream":"stdout","time":"2024-03-31T21:43:28.766545005Z"}
{"log":"2024-03-31 16:43:28.766113753  192.168.1.235 - - [31/Mar/2024:16:43:28 -0500] \"GET /ws HTTP/1.1\" 502 157 \"-\" \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0\" \"-\"\n","stream":"stdout","time":"2024-03-31T21:43:28.766638405Z"}
{"log":"2024-03-31 16:43:29.284450972  [INFO] Starting go2rtc...\n","stream":"stdout","time":"2024-03-31T21:43:29.284845711Z"}
{"log":"2024-03-31 16:43:29.483004620  16:43:29.482 INF go2rtc version 1.8.4 linux/amd64\n","stream":"stdout","time":"2024-03-31T21:43:29.483348302Z"}
{"log":"2024-03-31 16:43:29.484806072  16:43:29.484 INF [api] listen addr=:1984\n","stream":"stdout","time":"2024-03-31T21:43:29.485117655Z"}
{"log":"2024-03-31 16:43:29.484814378  16:43:29.484 INF [rtsp] listen addr=:8554\n","stream":"stdout","time":"2024-03-31T21:43:29.48516374Z"}
{"log":"2024-03-31 16:43:29.485437612  16:43:29.485 INF [webrtc] listen addr=:8555\n","stream":"stdout","time":"2024-03-31T21:43:29.487990094Z"}
{"log":"2024-03-31 16:43:32.873533618  [2024-03-31 16:43:32] frigate.app                    INFO    : Starting Frigate (0.13.2-6476f8a)

Operating system

Debian

Install method

Docker Compose

Coral version

CPU (no coral)

Any other information that may be helpful

No response

Answered by NickM-27

Mar 31, 2024

looks like you are running an old 0.12 model, you need to update your config for the 0.13 way of doing things

View full answer

NickM-27 · 2024-03-31T22:01:19Z

NickM-27
Mar 31, 2024
Collaborator Sponsor

looks like you are running an old 0.12 model, you need to update your config for the 0.13 way of doing things

0 replies

oldhome7 · 2024-03-31T22:47:25Z

oldhome7
Mar 31, 2024
Author

frigate@debian:~$ nvidia-smi
Sun Mar 31 17:46:34 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro K620                    On  |   00000000:01:00.0 Off |                  N/A |
| 34%   41C    P0              2W /   30W |      50MiB /   2048MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A    658325      C   ffmpeg                                         45MiB |
+-----------------------------------------------------------------------------------------+

0 replies

oldhome7 · 2024-03-31T22:51:15Z

oldhome7
Mar 31, 2024
Author

I just built this a couple days ago following the getting started guide, how would I be running a 12 model? Also, how can you tell that? I'm absolutely clueless, I just know how to follow directions.

I will say, I did attempt to start this up on Debian 12 a few weeks ago but, I ran into the nVidia driver issue and just decided to do it on Debian 11.

16 replies

oldhome7 Apr 1, 2024
Author

Still no frames received error

2024-03-31 19:35:01.580218417  [INFO] Preparing Frigate...
2024-03-31 19:35:01.646627678  [INFO] Starting Frigate...
2024-03-31 19:35:06.454632726  [2024-03-31 19:35:06] frigate.app                    INFO    : Starting Frigate (0.13.2-6476f8a)
2024-03-31 19:35:07.200106018  [2024-03-31 19:35:07] peewee_migrate.logs            INFO    : Starting migrations
2024-03-31 19:35:07.208758896  [2024-03-31 19:35:07] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-03-31 19:35:07.218954621  [2024-03-31 19:35:07] frigate.app                    INFO    : Recording process started: 451
2024-03-31 19:35:07.224414087  [2024-03-31 19:35:07] frigate.app                    INFO    : go2rtc process pid: 98
2024-03-31 19:35:07.268077029  [2024-03-31 19:35:07] frigate.app                    INFO    : Output process started: 461
2024-03-31 19:35:07.292113811  [2024-03-31 19:35:07] frigate.app                    INFO    : Camera processor started for camera1: 468
2024-03-31 19:35:07.294437953  [2024-03-31 19:35:07] frigate.app                    INFO    : Capture process started for camera1: 470
2024-03-31 19:35:07.297161239  [2024-03-31 19:35:07] detector.tensorrt              INFO    : Starting detection process: 460
2024-03-31 19:35:07.695457961  [2024-03-31 19:35:07] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 148 MiB
2024-03-31 19:35:08.296910582  [2024-03-31 19:35:08] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +8, now: CPU 268, GPU 182 (MiB)
2024-03-31 19:35:08.329157498  [2024-03-31 19:35:08] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 269, GPU 192 (MiB)
2024-03-31 19:35:08.351191867  [2024-03-31 19:35:08] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +148, now: CPU 0, GPU 148 (MiB)
2024-03-31 19:35:08.351379074  [2024-03-31 19:35:08] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +1, GPU +8, now: CPU 122, GPU 185 (MiB)
2024-03-31 19:35:08.357898794  [2024-03-31 19:35:08] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 122, GPU 193 (MiB)
2024-03-31 19:35:08.358008426  [2024-03-31 19:35:08] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +29, now: CPU 0, GPU 177 (MiB)
2024-03-31 19:35:13.834962232  Process camera_processor:camera1:
2024-03-31 19:35:13.837907434  Traceback (most recent call last):
2024-03-31 19:35:13.837972833    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-03-31 19:35:13.837977518      self.run()
2024-03-31 19:35:13.837981497    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-03-31 19:35:13.837990697      self._target(*self._args, **self._kwargs)
2024-03-31 19:35:13.837997670    File "/opt/frigate/frigate/video.py", line 436, in track_camera
2024-03-31 19:35:13.838038321      process_frames(
2024-03-31 19:35:13.838042740    File "/opt/frigate/frigate/video.py", line 689, in process_frames
2024-03-31 19:35:13.838046686      detect(
2024-03-31 19:35:13.838073380    File "/opt/frigate/frigate/video.py", line 474, in detect
2024-03-31 19:35:13.838077373      region_detections = object_detector.detect(tensor_input)
2024-03-31 19:35:13.838081155    File "/opt/frigate/frigate/object_detection.py", line 225, in detect
2024-03-31 19:35:13.838084818      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2024-03-31 19:35:13.838087968  KeyError: -10

oldhome7 Apr 1, 2024
Author

mqtt:
  enabled: False

detectors:
#  cpu:
#    type: cpu
#    num_threads: 6
#    model:
#      path: "/custom_model.tflite"

  tensorrt:
    type: tensorrt
    device: 0 #This is the default, select the first GPU

model:
  path: /config/model_cache/tensorrt/yolov7-320.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 320
  height: 320

ffmpeg:
  hwaccel_args: preset-nvidia-h264

cameras:
  camera1: # <--- this will be changed to your actual camera later
    enabled: True
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED:REDACTED@192.168.1.172:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
            - record
      output_args:
        record: preset-record-generic-audio-copy
    detect:
      enabled: true # <---- disable detection until you have a working camera feed
      width: 704 # <---- update for your camera's resolution
      height: 480 # <---- update for your camera's resolution

NickM-27 Apr 2, 2024
Collaborator Sponsor

ah yeah, this would be fixed by #10210

oldhome7 Apr 3, 2024
Author

So, do I just sit and wait for it to make it into the tensorrt and stable branches?

Also, I noticed if I restart the container, I will get an initial image on the camera screen, and no error message until I actually click onto the feed.

NickM-27 Apr 4, 2024
Collaborator Sponsor

You can always run the dev image if you choose to do so

oldhome7 · 2024-04-07T15:41:18Z

oldhome7
Apr 7, 2024
Author

How would I change to the dev image to test this? Just change my compose file from frigate:stable-tensorrt to frigate:dev I'm assuming?

13 replies

NickM-27 Apr 7, 2024
Collaborator Sponsor

you set the env var shown there in your docker compose or docker cli

oldhome7 Apr 7, 2024
Author

Again, noob here, what is the var name and attribute I need to set? I don't understand that line.

oldhome7 Apr 7, 2024
Author

I'm assuming?

  environment:
    - os.environ.get=true

Edit: nope, that's not it.

oldhome7 Apr 7, 2024
Author

#9877 says to add I_PROMISE_I_WONT_MAKE_AN_ISSUE_ON_GITHUB but, I'm not figuring out how to get that to work either.

oldhome7 Apr 7, 2024
Author

I got it, I was entering the var syntax incorrectly.

oldhome7 · 2024-04-07T23:49:40Z

oldhome7
Apr 7, 2024
Author

Well, it seems to be running well enough now. I'm blaming this on the puny K620 I'm running but, with the default YOLOv7-320 model I was getting about 55ms inference speeds. Switched it to the v7-tiny-418 and I'm down to about 20ms.

By all the maths I've seen, that should be good for about 10 cameras. This K620 should hold me over until I decide I need more cameras than that. I had plans for the dual TPU coral at some point, just wanted to play with hardware I've already got.

0 replies

toperichvania · 2024-11-08T08:38:38Z

toperichvania
Nov 8, 2024

I have similar issue on 0.14 however no additional logs are provided it just stops:

2024-11-08 08:34:41.417891763 [2024-11-08 08:34:41] frigate.app INFO : Capture process started for td_aux_space_39: 1408
2024-11-08 08:34:41.418550412 [2024-11-08 08:34:41] frigate.app INFO : Capture process not started for disabled camera PTZ
2024-11-08 08:34:44.521953712 08:34:44.521 WRN [rtsp] error="streams: exec: [tcp @ 0x5921c1163140] Connection to tcp://10.101.99.39:554?timeout=5000000 failed: No route to host\nrtsp://Frankentek:$PJ821$24@10.101.99.39/ONVIF/MediaInput?profile=def_profile1: No route to host\n" stream=td_aux_space_39
2024-11-08 08:34:44.523276780 [2024-11-08 08:34:44] frigate.video ERROR : td_aux_space_39: Unable to read frames from ffmpeg process.
2024-11-08 08:34:44.523525991 [2024-11-08 08:34:44] frigate.video ERROR : td_aux_space_39: ffmpeg process is not running. exiting capture thread...
2024-11-08 08:35:01.441696742 [2024-11-08 08:35:01] watchdog.td_aux_space_39 ERROR : Ffmpeg process crashed unexpectedly for td_aux_space_39.
2024-11-08 08:35:01.441741384 [2024-11-08 08:35:01] watchdog.td_aux_space_39 ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2024-11-08 08:35:01.441773134 [2024-11-08 08:35:01] ffmpeg.td_aux_space_39.detect ERROR : [rtsp @ 0x5a9e26001040] method DESCRIBE failed: 404 Not Found
2024-11-08 08:35:01.441799646 [2024-11-08 08:35:01] ffmpeg.td_aux_space_39.detect ERROR : rtsp://127.0.0.1:8554/td_aux_space_39: Server returned 404 Not Found
2024-11-08 08:35:01.548029861 [2024-11-08 08:35:01] frigate.watchdog INFO : Detection appears to have stopped. Exiting Frigate...
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service go2rtc-healthcheck: stopping

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

frigate.watchdog detection appears to have stopped #10764

{{title}}

Replies: 6 comments 29 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

frigate.watchdog detection appears to have stopped #10764

oldhome7 Mar 31, 2024

Describe the problem you are having

Version

Frigate config file

docker-compose file or Docker CLI command

Relevant log output

Operating system

Install method

Coral version

Any other information that may be helpful

Replies: 6 comments · 29 replies

NickM-27 Mar 31, 2024 Collaborator Sponsor

oldhome7 Mar 31, 2024 Author

oldhome7 Mar 31, 2024 Author

oldhome7 Apr 1, 2024 Author

oldhome7 Apr 1, 2024 Author

NickM-27 Apr 2, 2024 Collaborator Sponsor

oldhome7 Apr 3, 2024 Author

NickM-27 Apr 4, 2024 Collaborator Sponsor

oldhome7 Apr 7, 2024 Author

NickM-27 Apr 7, 2024 Collaborator Sponsor

oldhome7 Apr 7, 2024 Author

oldhome7 Apr 7, 2024 Author

oldhome7 Apr 7, 2024 Author

oldhome7 Apr 7, 2024 Author

oldhome7 Apr 7, 2024 Author

toperichvania Nov 8, 2024

oldhome7
Mar 31, 2024

Replies: 6 comments 29 replies

NickM-27
Mar 31, 2024
Collaborator Sponsor

oldhome7
Mar 31, 2024
Author

oldhome7
Mar 31, 2024
Author

oldhome7 Apr 1, 2024
Author

oldhome7 Apr 1, 2024
Author

NickM-27 Apr 2, 2024
Collaborator Sponsor

oldhome7 Apr 3, 2024
Author

NickM-27 Apr 4, 2024
Collaborator Sponsor

oldhome7
Apr 7, 2024
Author

NickM-27 Apr 7, 2024
Collaborator Sponsor

oldhome7 Apr 7, 2024
Author

oldhome7 Apr 7, 2024
Author

oldhome7 Apr 7, 2024
Author

oldhome7 Apr 7, 2024
Author

oldhome7
Apr 7, 2024
Author

toperichvania
Nov 8, 2024