Error while running `video_understanding` demo

[ruspaul013]

Hello!

While trying to run the video_understanding demo, I got this error:

Please input the video path:(Waiting for input. Your input can only be text or image path each time, you can press Enter once to input multiple times. Press Enter twice to finish the entire
 input.):
>>>/root/test.mov
>>>what is inside?
>>>
INFO | 2025-01-23 15:12:21,107 | /root/github/OmAgent/examples/video_understanding/run_cli.py:79 | <module> | User input lines: ['/root/test.mov', 'what is inside?']
INFO | 2025-01-23 15:12:21,457 | /root/github/OmAgent/omagent-core/src/omagent_core/clients/devices/app/input.py:79 | read_input | Received message: {'payload': '{"agent_id": "57921ba3-b050-4138-bac7-0cfd4b885553", "messages": [{"role": "user", "content": [{"type": "image_url", "data": "/root/test.mov"}, {"type": "text", "data": "what is inside?"}]}], "kwargs": {}}'}
  Detected: 0 | Progress:   0%|                                                                                                                                  | 0/582 [00:00<?, ?frames/s]2025-01-23 15:12:21,555 pyscenedetect INFO     Detecting scenes...
  Detected: 0 | Progress: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 581/582 [00:01<00:00, 567.87frames/s]2025-01-23 15:12:22,714 [3113034] omagent_core.engine.worker.base ERROR    Error executing task VideoPreprocessor with id de26c1ab-748a-40c6-a641-c4147b1c6b6c.  error = Traceback (most recent call last):
  File "/root/github/OmAgent/omagent-core/src/omagent_core/engine/worker/base.py", line 141, in execute
    task_output = self._run(**task_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/examples/video_understanding/agent/video_preprocessor/video_preprocess.py", line 182, in _run
    video = VideoScenes.load(
            ^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/examples/video_understanding/agent/misc/scene.py", line 98, in load
    audio = AudioSegment.from_file(video_path)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/pydub/audio_segment.py", line 728, in from_file
    info = mediainfo_json(orig_file, read_ahead_limit=read_ahead_limit)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/pydub/utils.py", line 279, in mediainfo_json
    info = json.loads(output)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
               ^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)

I am using local models and for STT and Embedding I followed the example from #197 .

Version: v0.2.2

If you need more details, please let me know.

[XeonHis]

It seems there are some audio issues in the video file.
Could you provide the related video file /root/test.mov so we can find out the reason further? In the meantime, you can try using other video files to test the video_understanding process again.

[ruspaul013]

Sorry but I can't give you the test.mov file, but I found another one with the same problem.

Please input the video path:(Waiting for input. Your input can only be text or image path each time, you can press Enter once to input multiple times. Press Enter twice to finish the entire
 input.):
>>>/nfs/exports/file_example_AVI_640_800kB.avi
>>>what is in this video?
>>>
INFO | 2025-01-24 09:39:47,393 | /root/github/OmAgent/examples/video_understanding/run_cli.py:79 | <module> | User input lines: ['/nfs/exports/file_example_AVI_640_800kB.avi', 'what is in t
his video?']
INFO | 2025-01-24 09:39:47,977 | /root/github/OmAgent/omagent-core/src/omagent_core/clients/devices/app/input.py:79 | read_input | Received message: {'payload': '{"agent_id": "e77f4dd4-0d11
-495c-9df0-a64148a71090", "messages": [{"role": "user", "content": [{"type": "image_url", "data": "/nfs/exports/file_example_AVI_640_800kB.avi"}, {"type": "text", "data": "what is in this v
ideo?"}]}], "kwargs": {}}'}
  Detected: 0 | Progress:   0%|                                                                                                                                  | 0/901 [00:00<?, ?frames/s]
2025-01-24 09:39:48,033 pyscenedetect INFO     Detecting scenes...
  Detected: 0 | Progress: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 901/901 [00:00<00:00, 1397.57frames/s]
2025-01-24 09:39:48,826 [3404590] omagent_core.engine.worker.base ERROR    Error executing task VideoPreprocessor with id 25ace73f-c30f-4cff-acc4-23114e16ed18.  error = Traceback (most rece
nt call last):
  File "/root/github/OmAgent/omagent-core/src/omagent_core/engine/worker/base.py", line 141, in execute
    task_output = self._run(**task_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/examples/video_understanding/agent/video_preprocessor/video_preprocess.py", line 182, in _run
    video = VideoScenes.load(
            ^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/examples/video_understanding/agent/misc/scene.py", line 98, in load
    audio = AudioSegment.from_file(video_path)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/pydub/audio_segment.py", line 728, in from_file
    info = mediainfo_json(orig_file, read_ahead_limit=read_ahead_limit)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/pydub/utils.py", line 279, in mediainfo_json
    info = json.loads(output)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
               ^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)

file: https://file-examples.com/index.php/sample-video-files/sample-avi-files-download/ - the second file ( I could not attach the avi file )

[ruspaul013]

I tried with a wmv file, but I get this error

Please input the video path:(Waiting for input. Your input can only be text or image path each time, you can press Enter once to input multiple times. Press Enter twice to finish the entire
 input.):
>>>/nfs/exports/file_example_WMV_640_1_6MB.wmv
>>>what is in this video?
>>>
INFO | 2025-01-24 09:41:29,675 | /root/github/OmAgent/examples/video_understanding/run_cli.py:79 | <module> | User input lines: ['/nfs/exports/file_example_WMV_640_1_6MB.wmv', 'what is in t
his video?']
INFO | 2025-01-24 09:41:30,128 | /root/github/OmAgent/omagent-core/src/omagent_core/clients/devices/app/input.py:79 | read_input | Received message: {'payload': '{"agent_id": "e77f4dd4-0d11
-495c-9df0-a64148a71090", "messages": [{"role": "user", "content": [{"type": "image_url", "data": "/nfs/exports/file_example_WMV_640_1_6MB.wmv"}, {"type": "text", "data": "what is in this v
ideo?"}]}], "kwargs": {}}'}
  Detected: 0 | Progress:   0%|                                                                                                                                  | 0/918 [00:00<?, ?frames/s]
2025-01-24 09:41:30,166 pyscenedetect INFO     Detecting scenes...
  Detected: 0 | Progress:  98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊  | 901/918 [00:00<00:00, 1535.27frames/s]
Processing scene 0 / 1...
2025-01-24 09:41:31,454 httpx        INFO     HTTP Request: POST http://10.150.92.131:8081/audio/transcriptions "HTTP/1.1 404 Not Found"
2025-01-24 09:41:31,465 [3404590] omagent_core.engine.worker.base ERROR    Error executing task VideoPreprocessor with id 974e0af5-39a8-4d36-a762-8deb4c7282e8.  error = Traceback (most rece
nt call last):
  File "/root/github/OmAgent/omagent-core/src/omagent_core/engine/worker/base.py", line 141, in execute
    task_output = self._run(**task_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/examples/video_understanding/agent/video_preprocessor/video_preprocess.py", line 198, in _run
    scene.stt_res = self.stt.infer(audio_clip)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/omagent-core/src/omagent_core/models/asr/stt.py", line 38, in infer
    trans = self.client.audio.transcriptions.create(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/openai/resources/audio/transcriptions.py", line 188, in create
    return self._post(  # type: ignore[return-value]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/openai/_base_client.py", line 1283, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/openai/_base_client.py", line 960, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/openai/_base_client.py", line 1064, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: Error code: 404 - {'error': {'code': 404, 'message': 'Resource not found', 'type': ''}}

and in local-ai I get

9:41AM WRN Client error ip=10.150.92.131 latency="361.705µs" method=POST status=404 url=/audio/transcriptions

I restarted the local-ai server and I get this now

Please input the video path:(Waiting for input. Your input can only be text or image path each time, you can press Enter once to input multiple times. Press Enter twice to finish the entire
 input.):
>>>/nfs/exports/file_example_WMV_640_1_6MB.wmv
>>>what is inside?
>>>
INFO | 2025-01-24 09:50:24,749 | /root/github/OmAgent/examples/video_understanding/run_cli.py:79 | <module> | User input lines: ['/nfs/exports/file_example_WMV_640_1_6MB.wmv', 'what is insi
de?']
INFO | 2025-01-24 09:50:25,253 | /root/github/OmAgent/omagent-core/src/omagent_core/clients/devices/app/input.py:79 | read_input | Received message: {'payload': '{"agent_id": "e77f4dd4-0d1$
-495c-9df0-a64148a71090", "messages": [{"role": "user", "content": [{"type": "image_url", "data": "/nfs/exports/file_example_WMV_640_1_6MB.wmv"}, {"type": "text", "data": "what is inside?"}
]}], "kwargs": {}}'}
2025-01-24 09:50:25,302 [3404590] omagent_core.engine.worker.base ERROR    Error executing task VideoPreprocessor with id b9b49f5d-ed3d-443e-86ac-fe85244efbd3.  error = Traceback (most rece
nt call last):
  File "/root/github/OmAgent/omagent-core/src/omagent_core/engine/worker/base.py", line 141, in execute
    task_output = self._run(**task_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/examples/video_understanding/agent/video_preprocessor/video_preprocess.py", line 251, in _run
    pickle.dump(video.scenes, f)
                ^^^^^
UnboundLocalError: cannot access local variable 'video' where it is not associated with a value

2025-01-24 09:50:29,813 [3404590] omagent_core.engine.automator.task_handler INFO     Stopped worker processes...

or

Please input the video path:(Waiting for input. Your input can only be text or image path each time, you can press Enter once to input multiple times. Press Enter twice to finish the entire input.):
>>>/nfs/exports/file_example_WMV_640_1_6MB.wmv
>>>what is inside this video?
>>>
INFO | 2025-01-24 09:55:54,651 | /root/github/OmAgent/examples/video_understanding/run_cli.py:79 | <module> | User input lines: ['/nfs/exports/file_example_WMV_640_1_6MB.wmv', 'what is inside this video?']
INFO | 2025-01-24 09:55:54,838 | /root/github/OmAgent/omagent-core/src/omagent_core/clients/devices/app/input.py:79 | read_input | Received message: {'payload': '{"agent_id": "3f77d919-c6a2-4fd5-924f-4b3f0a1abec9", "messages": [{"role": "user", "content": [{"type": "image_url", "data": "/nfs/exports/file_example_WMV_640_1_6MB.wmv"}, {"type": "text", "data": "what is inside this video?"}]}], "kwargs": {}}'}
2025-01-24 09:55:54,859 [3411265] omagent_core.engine.worker.base ERROR    Error executing task VideoPreprocessor with id e4cc7763-e4bf-443d-b344-a8be6e5bca7b.  error = Traceback (most recent call last):
  File "/root/github/OmAgent/omagent-core/src/omagent_core/engine/worker/base.py", line 141, in execute
    task_output = self._run(**task_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/examples/video_understanding/agent/video_preprocessor/video_preprocess.py", line 100, in _run
    loaded_scene = pickle.load(f)
                   ^^^^^^^^^^^^^^
EOFError: Ran out of input

file: https://file-examples.com/index.php/sample-video-files/sample-wmv-files-download/ - the second one

[yileld]

I tried with a wmv file, but I get this error

Please input the video path:(Waiting for input. Your input can only be text or image path each time, you can press Enter once to input multiple times. Press Enter twice to finish the entire
 input.):
>>>/nfs/exports/file_example_WMV_640_1_6MB.wmv
>>>what is in this video?
>>>
INFO | 2025-01-24 09:41:29,675 | /root/github/OmAgent/examples/video_understanding/run_cli.py:79 | <module> | User input lines: ['/nfs/exports/file_example_WMV_640_1_6MB.wmv', 'what is in t
his video?']
INFO | 2025-01-24 09:41:30,128 | /root/github/OmAgent/omagent-core/src/omagent_core/clients/devices/app/input.py:79 | read_input | Received message: {'payload': '{"agent_id": "e77f4dd4-0d11
-495c-9df0-a64148a71090", "messages": [{"role": "user", "content": [{"type": "image_url", "data": "/nfs/exports/file_example_WMV_640_1_6MB.wmv"}, {"type": "text", "data": "what is in this v
ideo?"}]}], "kwargs": {}}'}
  Detected: 0 | Progress:   0%|                                                                                                                                  | 0/918 [00:00<?, ?frames/s]
2025-01-24 09:41:30,166 pyscenedetect INFO     Detecting scenes...
  Detected: 0 | Progress:  98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊  | 901/918 [00:00<00:00, 1535.27frames/s]
Processing scene 0 / 1...
2025-01-24 09:41:31,454 httpx        INFO     HTTP Request: POST http://10.150.92.131:8081/audio/transcriptions "HTTP/1.1 404 Not Found"
2025-01-24 09:41:31,465 [3404590] omagent_core.engine.worker.base ERROR    Error executing task VideoPreprocessor with id 974e0af5-39a8-4d36-a762-8deb4c7282e8.  error = Traceback (most rece
nt call last):
  File "/root/github/OmAgent/omagent-core/src/omagent_core/engine/worker/base.py", line 141, in execute
    task_output = self._run(**task_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/examples/video_understanding/agent/video_preprocessor/video_preprocess.py", line 198, in _run
    scene.stt_res = self.stt.infer(audio_clip)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/omagent-core/src/omagent_core/models/asr/stt.py", line 38, in infer
    trans = self.client.audio.transcriptions.create(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/openai/resources/audio/transcriptions.py", line 188, in create
    return self._post(  # type: ignore[return-value]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/openai/_base_client.py", line 1283, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/openai/_base_client.py", line 960, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/openai/_base_client.py", line 1064, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: Error code: 404 - {'error': {'code': 404, 'message': 'Resource not found', 'type': ''}}

and in local-ai I get

9:41AM WRN Client error ip=10.150.92.131 latency="361.705µs" method=POST status=404 url=/audio/transcriptions

I restarted the local-ai server and I get this now

Please input the video path:(Waiting for input. Your input can only be text or image path each time, you can press Enter once to input multiple times. Press Enter twice to finish the entire
 input.):
>>>/nfs/exports/file_example_WMV_640_1_6MB.wmv
>>>what is inside?
>>>
INFO | 2025-01-24 09:50:24,749 | /root/github/OmAgent/examples/video_understanding/run_cli.py:79 | <module> | User input lines: ['/nfs/exports/file_example_WMV_640_1_6MB.wmv', 'what is insi
de?']
INFO | 2025-01-24 09:50:25,253 | /root/github/OmAgent/omagent-core/src/omagent_core/clients/devices/app/input.py:79 | read_input | Received message: {'payload': '{"agent_id": "e77f4dd4-0d1$
-495c-9df0-a64148a71090", "messages": [{"role": "user", "content": [{"type": "image_url", "data": "/nfs/exports/file_example_WMV_640_1_6MB.wmv"}, {"type": "text", "data": "what is inside?"}
]}], "kwargs": {}}'}
2025-01-24 09:50:25,302 [3404590] omagent_core.engine.worker.base ERROR    Error executing task VideoPreprocessor with id b9b49f5d-ed3d-443e-86ac-fe85244efbd3.  error = Traceback (most rece
nt call last):
  File "/root/github/OmAgent/omagent-core/src/omagent_core/engine/worker/base.py", line 141, in execute
    task_output = self._run(**task_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/examples/video_understanding/agent/video_preprocessor/video_preprocess.py", line 251, in _run
    pickle.dump(video.scenes, f)
                ^^^^^
UnboundLocalError: cannot access local variable 'video' where it is not associated with a value

2025-01-24 09:50:29,813 [3404590] omagent_core.engine.automator.task_handler INFO     Stopped worker processes...

or

Please input the video path:(Waiting for input. Your input can only be text or image path each time, you can press Enter once to input multiple times. Press Enter twice to finish the entire input.):
>>>/nfs/exports/file_example_WMV_640_1_6MB.wmv
>>>what is inside this video?
>>>
INFO | 2025-01-24 09:55:54,651 | /root/github/OmAgent/examples/video_understanding/run_cli.py:79 | <module> | User input lines: ['/nfs/exports/file_example_WMV_640_1_6MB.wmv', 'what is inside this video?']
INFO | 2025-01-24 09:55:54,838 | /root/github/OmAgent/omagent-core/src/omagent_core/clients/devices/app/input.py:79 | read_input | Received message: {'payload': '{"agent_id": "3f77d919-c6a2-4fd5-924f-4b3f0a1abec9", "messages": [{"role": "user", "content": [{"type": "image_url", "data": "/nfs/exports/file_example_WMV_640_1_6MB.wmv"}, {"type": "text", "data": "what is inside this video?"}]}], "kwargs": {}}'}
2025-01-24 09:55:54,859 [3411265] omagent_core.engine.worker.base ERROR    Error executing task VideoPreprocessor with id e4cc7763-e4bf-443d-b344-a8be6e5bca7b.  error = Traceback (most recent call last):
  File "/root/github/OmAgent/omagent-core/src/omagent_core/engine/worker/base.py", line 141, in execute
    task_output = self._run(**task_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/examples/video_understanding/agent/video_preprocessor/video_preprocess.py", line 100, in _run
    loaded_scene = pickle.load(f)
                   ^^^^^^^^^^^^^^
EOFError: Ran out of input

file: https://file-examples.com/index.php/sample-video-files/sample-wmv-files-download/ - the second one

I can run the avi file, and local-ai is

INF Success ip=127.0.0.1 latency=12.056258541s method=POST status=200 url=/v1/audio/transcriptions

My version of pydub is 0.25.1.
It seems you miss /v1 in video_preprocessor.yml

endpoint: http://localhost:8080/v1

And please update ./agent/conclude/conclude.py from other example /examples/general_dnc/agent/conclude/conclude.py about iterator judgement part. This fix isn't updated yet.

from collections.abc import Iterator
.
.
.
 if isinstance(chat_complete_res, Iterator):
     last_output = "Answer: "
     self.callback.send_incomplete(
         agent_id=self.workflow_instance_id, msg="Answer: "
     )
     for chunk in chat_complete_res:
         if len(chunk.choices) > 0:
             current_msg = chunk.choices[0].delta.content if chunk.choices[0].delta.content is not None else ''
             self.callback.send_incomplete(
                 agent_id=self.workflow_instance_id,
                 msg=f"{current_msg}",
             )
             last_output += current_msg
     self.callback.send_answer(agent_id=self.workflow_instance_id, msg="")
 else:
     last_output = chat_complete_res["choices"][0]["message"]["content"]
     self.callback.send_answer(
         agent_id=self.workflow_instance_id,
         msg=f'Answer: {chat_complete_res["choices"][0]["message"]["content"]}',
     )
 self.callback.send_answer(agent_id=self.workflow_instance_id, msg=f"Token usage: {self.token_usage}")
 self.stm(self.workflow_instance_id).clear()
 return {"last_output": last_output}

[ruspaul013]

Hello! I modified the utils.py from pydub as in the main branch to avoid the error. ( only some videos have this problem )

I was able to upload the video and ask a question but know I get:

info:3938c258-4b97-410a-a730-ac64a70b35a2 Conclude what can you see in the video?
2025-01-28 09:11:29,393 httpx        INFO     HTTP Request: POST https://devai.4psa.me/llm/v1/chat/completions "HTTP/1.1 200 OK"
2025-01-28 09:11:29,396 [827968] omagent_core.engine.worker.base ERROR    Error executing task Conclude with id e94f6d90-f44f-4624-9a71-8d11ef55ddc5.  error = Traceback (most recent call last):
  File "/root/github/OmAgent/omagent-core/src/omagent_core/engine/worker/base.py", line 141, in execute
    task_output = self._run(**task_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/examples/video_understanding/agent/conclude/conclude.py", line 83, in _run
    last_output = chat_complete_res["choices"][0]["message"]["content"]
                  ~~~~~~~~~~~~~~~~~^^^^^^^^^^^
TypeError: 'generator' object is not subscriptable

full log: full_log.txt
Version: v0.2.2

[ruspaul013]

Using v0.2.3 I get:

Please input the video path:(Waiting for input. Your input can only be text or image path each time, you can press Enter once to input multiple times. Press Enter twice to finish the entire input.):
>>>/nfs/exports/Venice_5.mp4
>>>
INFO | 2025-01-28 09:27:56,215 | /root/github/OmAgent/examples/video_understanding/run_cli.py:79 | <module> | User input lines: ['/nfs/exports/Venice_5.mp4']
INFO | 2025-01-28 09:27:56,302 | /root/github/OmAgent/omagent-core/src/omagent_core/clients/devices/app/input.py:79 | read_input | Received message: {'payload': '{"agent_id": "d26c93f1-dc88-4fe5-b404-30f9b5fb8235", "messages": [{"role":
"user", "content": [{"type": "image_url", "data": "/nfs/exports/Venice_5.mp4"}]}], "kwargs": {}}'}
  Detected: 0 | Progress:   0%|                                                                                                                                                                                  | 0/151 [00:00<?, ?frames/s]
2025-01-28 09:27:56,425 pyscenedetect INFO     Detecting scenes...
  Detected: 0 | Progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 151/151 [00:00<00:00, 234.83frames/s]
Processing scene 0 / 1...
2025-01-28 09:28:04,938 httpx        INFO     HTTP Request: POST http://10.150.92.131:8081/v1/audio/transcriptions "HTTP/1.1 200 OK"
2025-01-28 09:28:05,569 geocoder.base INFO     Requested http://ipinfo.io/json
2025-01-28 09:28:24,673 httpx        INFO     HTTP Request: POST https://devai.4psa.me/llm/v1/chat/completions "HTTP/1.1 200 OK"
2025-01-28 09:28:34,460 httpx        INFO     HTTP Request: POST http://10.150.92.131:8081/v1/embeddings "HTTP/1.1 200 OK"
INFO | 2025-01-28 09:28:38,689 | /root/github/OmAgent/omagent-core/src/omagent_core/engine/automator/task_runner.py:69 | run_once | Executing task, id: 0ce234c0-ccf5-4c63-bc5c-b553c954ee77, workflow_instance_id: d26c93f1-dc88-4fe5-b404-3
0f9b5fb8235, task_definition_name: VideoQA
INFO | 2025-01-28 09:28:38,707 | /root/github/OmAgent/omagent-core/src/omagent_core/clients/devices/app/input.py:244 | _send_base_message | Stream: d26c93f1-dc88-4fe5-b404-30f9b5fb8235_output, Group: omappagent, Data: {'payload': '{"agen
t_id": "d26c93f1-dc88-4fe5-b404-30f9b5fb8235", "code": 0, "error_info": "", "took": 0, "content_status": "end_block", "interaction_type": 1, "message": {"role": "assistant", "type": "text", "content": "Please input your question:"}, "usa
ge": {"prompt_tokens": 0, "output_tokens": 0}}'}
INFO | 2025-01-28 09:28:38,727 | /root/github/OmAgent/examples/video_understanding/agent/video_qa/qa.py:39 | _run | Listening to Redis stream: d26c93f1-dc88-4fe5-b404-30f9b5fb8235_input in group: omappagent start_id: b'1738049318726-0'
INFO | 2025-01-28 09:28:39,531 | /root/github/OmAgent/omagent-core/src/omagent_core/clients/devices/cli/client.py:124 | start_interactor | Received message: {'payload': '{"agent_id": "d26c93f1-dc88-4fe5-b404-30f9b5fb8235", "code": 0, "er
ror_info": "", "took": 0, "content_status": "end_block", "interaction_type": 1, "message": {"role": "assistant", "type": "text", "content": "Please input your question:"}, "usage": {"prompt_tokens": 0, "output_tokens": 0}}'}
Please input your question:(Waiting for input. Your input can only be text or image path each time, you can press Enter once to input multiple times. Press Enter twice to finish the entire input.):
>>>take some details from the video
>>>
INFO | 2025-01-28 09:29:02,304 | /root/github/OmAgent/examples/video_understanding/run_cli.py:79 | <module> | User input lines: ['take some details from the video']
INFO | 2025-01-28 09:29:02,846 | /root/github/OmAgent/omagent-core/src/omagent_core/clients/devices/app/input.py:79 | read_input | Received message: {'payload': '{"agent_id": "d26c93f1-dc88-4fe5-b404-30f9b5fb8235", "messages": [{"role":
"user", "content": [{"type": "text", "data": "take some details from the video"}]}], "kwargs": {}}'}
2025-01-28 09:29:03,061 geocoder.base INFO     Requested http://ipinfo.io/json
2025-01-28 09:29:09,309 httpx        INFO     HTTP Request: POST https://devai.4psa.me/llm/v1/chat/completions "HTTP/1.1 200 OK"
2025-01-28 09:29:09,637 httpx        INFO     HTTP Request: POST http://10.150.92.131:8081/v1/embeddings "HTTP/1.1 200 OK"
INFO | 2025-01-28 09:29:14,059 | /root/github/OmAgent/omagent-core/src/omagent_core/engine/automator/task_runner.py:69 | run_once | Executing task, id: 1e274b1a-b85c-4f10-8675-f8ad07d8e123, workflow_instance_id: d26c93f1-dc88-4fe5-b404-3
0f9b5fb8235, task_definition_name: ConstructDncPayload
INFO | 2025-01-28 09:29:20,148 | /root/github/OmAgent/omagent-core/src/omagent_core/engine/automator/task_runner.py:69 | run_once | Executing task, id: 2fb3f3a8-9f56-4521-9ade-444ccc56e832, workflow_instance_id: d26c93f1-dc88-4fe5-b404-3
0f9b5fb8235, task_definition_name: StructureUpdate
INFO | 2025-01-28 09:29:24,198 | /root/github/OmAgent/omagent-core/src/omagent_core/engine/automator/task_runner.py:69 | run_once | Executing task, id: 42d6c466-4927-408d-a15c-2b1791df28d7, workflow_instance_id: d26c93f1-dc88-4fe5-b404-3
0f9b5fb8235, task_definition_name: TaskConqueror
2025-01-28 09:29:24,472 geocoder.base INFO     Requested http://ipinfo.io/json
2025-01-28 09:29:24,553 httpx        INFO     HTTP Request: POST https://devai.4psa.me/llm/v1/chat/completions "HTTP/1.1 400 BAD REQUEST"
2025-01-28 09:29:24,560 [836044] omagent_core.engine.worker.base ERROR    Error executing task TaskConqueror with id 42d6c466-4927-408d-a15c-2b1791df28d7.  error = Traceback (most recent call last):
  File "/root/github/OmAgent/omagent-core/src/omagent_core/engine/worker/base.py", line 141, in execute
    task_output = self._run(**task_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/omagent-core/src/omagent_core/advanced_components/workflow/dnc/agent/conqueror/conqueror.py", line 104, in _run
    chat_complete_res = self.infer(input_list=[payload])
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/omagent-core/src/omagent_core/models/llms/base.py", line 220, in infer
    output = self.llm.generate(prompt, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/omagent-core/src/omagent_core/models/llms/base.py", line 63, in generate
    return self._call(records, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/omagent-core/src/omagent_core/models/llms/openai_gpt.py", line 132, in _call
    res = self.client.chat.completions.create(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/openai/_utils/_utils.py", line 279, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/openai/resources/chat/completions.py", line 850, in create
    return self._post(
           ^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/openai/_base_client.py", line 1283, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/openai/_base_client.py", line 960, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/root/omagent/lib64/python3.11/site-packages/openai/_base_client.py", line 1064, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'detail': '500: Internal Server Error'}

[XeonHis]

2025-01-28 09:29:24,553 httpx INFO HTTP Request: POST https://devai.4psa.me/llm/v1/chat/completions "HTTP/1.1 400 BAD REQUEST"

According to this log, it seems your custom API provider has some error while handling LLM requests.

[ruspaul013]

2025-01-28 09:29:24,553 httpx INFO HTTP Request: POST https://devai.4psa.me/llm/v1/chat/completions "HTTP/1.1 400 BAD REQUEST"

According to this log, it seems your custom API provider has some error while handling LLM requests.

Yes, it seems that slang return some errors.

L.E: This is the error:

  File "/usr/local/lib/python3.10/dist-packages/pydantic/main.py", line 214, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 2 validation errors for ChatCompletionRequest
tool_choice.ToolChoice
  Input should be a valid dictionary or instance of ToolChoice [type=model_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.10/v/model_type
tool_choice.literal['auto','required','none']
  Input should be 'auto', 'required' or 'none' [type=literal_error, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error

I saw that here, default value is None. If I change it to "none", I get the same error from version v0.2.2.

[ruspaul013]

Hello! I modified the utils.py from pydub as in the main branch to avoid the error. ( only some videos have this problem )

I was able to upload the video and ask a question but know I get:

info:3938c258-4b97-410a-a730-ac64a70b35a2 Conclude what can you see in the video?
2025-01-28 09:11:29,393 httpx        INFO     HTTP Request: POST https://devai.4psa.me/llm/v1/chat/completions "HTTP/1.1 200 OK"
2025-01-28 09:11:29,396 [827968] omagent_core.engine.worker.base ERROR    Error executing task Conclude with id e94f6d90-f44f-4624-9a71-8d11ef55ddc5.  error = Traceback (most recent call last):
  File "/root/github/OmAgent/omagent-core/src/omagent_core/engine/worker/base.py", line 141, in execute
    task_output = self._run(**task_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/github/OmAgent/examples/video_understanding/agent/conclude/conclude.py", line 83, in _run
    last_output = chat_complete_res["choices"][0]["message"]["content"]
                  ~~~~~~~~~~~~~~~~~^^^^^^^^^^^
TypeError: 'generator' object is not subscriptable

full log: full_log.txt Version: v0.2.2

It seems that I did not used a multimodal model to have support for vision.