Skip to content

Docker: Fix video recording in Node consume high CPU #2856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 7, 2025
Merged

Conversation

VietND96
Copy link
Member

@VietND96 VietND96 commented Jun 6, 2025

User description

Thanks for contributing to the Docker-Selenium project!
A PR well described will help maintainers to quickly review and merge it

Before submitting your PR, please check our contributing guidelines, applied for this repository.
Avoid large PRs, help reviewers by making them as simple and short as possible.

Description

Fixes #2788

Motivation and Context

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • I have read the contributing document.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

PR Type

Bug fix, Enhancement


Description

  • Optimize ffmpeg video recording to reduce Node CPU usage

    • Adjust ffmpeg parameters for efficiency and configurability
    • Lower default ffmpeg thread count and tune encoding settings
  • Improve video recording polling and logging logic

    • Increase poll interval and add clearer waiting log messages
  • Update test Docker Compose files for video recording

    • Enable video recording in Node services by default
    • Increase Node replicas for Chrome and Firefox in get-started config
  • Set browser download capability to false in tests


Changes walkthrough 📝

Relevant files
Enhancement
video.sh
Optimize ffmpeg usage and polling for video recording       

Video/video.sh

  • Refactored ffmpeg command to use more efficient parameters and
    environment variables.
  • Reduced default ffmpeg thread count and added encoding optimizations.
  • Increased polling interval for video recording checks.
  • Improved logging and waiting logic when no session is present.
  • +14/-5   
    get_started.py
    Disable browser downloads in test setup                                   

    tests/get_started.py

  • Set browser download capability (enable_downloads) to False for all
    browsers.
  • +1/-1     
    Configuration changes
    docker-compose-v3-dev-arm64.yml
    Enable video recording in dev ARM64 Docker Compose             

    tests/docker-compose-v3-dev-arm64.yml

  • Enabled video recording (SE_RECORD_VIDEO=true) for Chrome and Firefox
    Node services.
  • +2/-0     
    docker-compose-v3-get-started-arm64.yml
    Scale up Node replicas and clean up ports in get-started config

    tests/docker-compose-v3-get-started-arm64.yml

  • Increased Chrome and Firefox Node replicas from 1 to 3.
  • Removed VNC port mappings for both browsers.
  • +2/-6     

    Need help?
  • Type /help how to ... in the comments thread for any questions about Qodo Merge usage.
  • Check out the documentation for more information.
  • Copy link

    qodo-merge-pro bot commented Jun 6, 2025

    PR Reviewer Guide 🔍

    Here are some key observations to aid the review process:

    ⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
    🧪 No relevant tests
    🔒 No security concerns identified
    ⚡ Recommended focus areas for review

    Error Handling

    The ffmpeg command at line 268-272 is missing error handling. If ffmpeg fails to start, the FFMPEG_PID variable will be empty but the code assumes it's valid. This could lead to unexpected behavior.

    ffmpeg -hide_banner -loglevel warning -threads ${SE_FFMPEG_THREADS:-1} -thread_queue_size 512 \
    -probesize 32M -analyzeduration 0 -y -f x11grab -video_size ${VIDEO_SIZE} -r ${FRAME_RATE} \
    -i ${DISPLAY} ${SE_AUDIO_SOURCE} -codec:v ${CODEC} ${PRESET:-"-preset veryfast"} \
    -tune zerolatency -crf ${SE_VIDEO_CRF:-28} -maxrate ${SE_VIDEO_MAXRATE:-1000k} -bufsize ${SE_VIDEO_BUFSIZE:-2000k} \
    -pix_fmt yuv420p -movflags +faststart "$video_file" &
    Random Sleep

    The random sleep duration (RANDOM % 16 + poll_interval) might cause unpredictable behavior. Consider using a fixed or configurable value instead of a random one for more consistent performance.

      sleep $((RANDOM % 16 + poll_interval))
    fi

    Copy link

    qodo-merge-pro bot commented Jun 6, 2025

    CI Feedback 🧐

    (Feedback updated until commit 047d18c)

    A test triggered by this PR failed. Here is an AI-generated analysis of the failure:

    Action: Test Selenium Grid on Docker / Test Docker Selenium (test_video, false, true, false, ubuntu-24.04)

    Failed stage: Format and lint scripts [❌]

    Failed test name: format_shell_scripts

    Failure summary:

    The action failed because shell scripts in the repository are not properly formatted. The error
    occurred during the execution of the format_shell_scripts target in the Makefile (line 77). The
    system detected formatting issues in the Video/video.sh file and instructed to run 'make
    format_shell_scripts' to fix the formatting.

    Relevant error logs:
    1:  ##[group]Operating System
    2:  Ubuntu
    ...
    
    148:  �[36;1m�[0m
    149:  �[36;1m  sudo rm -rf /opt/ghc || true�[0m
    150:  �[36;1m  sudo rm -rf /usr/local/.ghcup || true�[0m
    151:  �[36;1m  �[0m
    152:  �[36;1m  AFTER=$(getAvailableSpace)�[0m
    153:  �[36;1m  SAVED=$((AFTER-BEFORE))�[0m
    154:  �[36;1m  printSavedSpace $SAVED "Haskell runtime"�[0m
    155:  �[36;1mfi�[0m
    156:  �[36;1m�[0m
    157:  �[36;1m# Option: Remove large packages�[0m
    158:  �[36;1m# REF: https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh�[0m
    159:  �[36;1m�[0m
    160:  �[36;1mif [[ false == 'true' ]]; then�[0m
    161:  �[36;1m  BEFORE=$(getAvailableSpace)�[0m
    162:  �[36;1m  �[0m
    163:  �[36;1m  sudo apt-get remove -y '^aspnetcore-.*' || echo "::warning::The command [sudo apt-get remove -y '^aspnetcore-.*'] failed to complete successfully. Proceeding..."�[0m
    164:  �[36;1m  sudo apt-get remove -y '^dotnet-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^dotnet-.*' --fix-missing] failed to complete successfully. Proceeding..."�[0m
    165:  �[36;1m  sudo apt-get remove -y '^llvm-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^llvm-.*' --fix-missing] failed to complete successfully. Proceeding..."�[0m
    166:  �[36;1m  sudo apt-get remove -y 'php.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y 'php.*' --fix-missing] failed to complete successfully. Proceeding..."�[0m
    167:  �[36;1m  sudo apt-get remove -y '^mongodb-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^mongodb-.*' --fix-missing] failed to complete successfully. Proceeding..."�[0m
    168:  �[36;1m  sudo apt-get remove -y '^mysql-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^mysql-.*' --fix-missing] failed to complete successfully. Proceeding..."�[0m
    169:  �[36;1m  sudo apt-get remove -y azure-cli google-chrome-stable firefox powershell mono-devel libgl1-mesa-dri --fix-missing || echo "::warning::The command [sudo apt-get remove -y azure-cli google-chrome-stable firefox powershell mono-devel libgl1-mesa-dri --fix-missing] failed to complete successfully. Proceeding..."�[0m
    170:  �[36;1m  sudo apt-get remove -y google-cloud-sdk --fix-missing || echo "::debug::The command [sudo apt-get remove -y google-cloud-sdk --fix-missing] failed to complete successfully. Proceeding..."�[0m
    171:  �[36;1m  sudo apt-get remove -y google-cloud-cli --fix-missing || echo "::debug::The command [sudo apt-get remove -y google-cloud-cli --fix-missing] failed to complete successfully. Proceeding..."�[0m
    172:  �[36;1m  sudo apt-get autoremove -y || echo "::warning::The command [sudo apt-get autoremove -y] failed to complete successfully. Proceeding..."�[0m
    173:  �[36;1m  sudo apt-get clean || echo "::warning::The command [sudo apt-get clean] failed to complete successfully. Proceeding..."�[0m
    174:  �[36;1m�[0m
    ...
    
    489:  ffmpeg -hide_banner -loglevel warning -threads ${SE_FFMPEG_THREADS:-1} -thread_queue_size 512 \
    490:  -        -probesize 32M -analyzeduration 0 -y -f x11grab -video_size ${VIDEO_SIZE} -r ${FRAME_RATE} \
    491:  -        -i ${DISPLAY} ${SE_AUDIO_SOURCE} -codec:v ${CODEC} ${PRESET:-"-preset veryfast"} \
    492:  -        -tune zerolatency -crf ${SE_VIDEO_CRF:-28} -maxrate ${SE_VIDEO_MAXRATE:-1000k} -bufsize ${SE_VIDEO_BUFSIZE:-2000k} \
    493:  -        -pix_fmt yuv420p -movflags +faststart "$video_file" &
    494:  +          -probesize 32M -analyzeduration 0 -y -f x11grab -video_size ${VIDEO_SIZE} -r ${FRAME_RATE} \
    495:  +          -i ${DISPLAY} ${SE_AUDIO_SOURCE} -codec:v ${CODEC} ${PRESET:-"-preset veryfast"} \
    496:  +          -tune zerolatency -crf ${SE_VIDEO_CRF:-28} -maxrate ${SE_VIDEO_MAXRATE:-1000k} -bufsize ${SE_VIDEO_BUFSIZE:-2000k} \
    497:  +          -pix_fmt yuv420p -movflags +faststart "$video_file" &
    498:  FFMPEG_PID=$!
    499:  if ps -p $FFMPEG_PID >/dev/null; then
    500:  recording_started="true"
    501:  Video/video.sh | 16 ++++++++--------
    502:  1 file changed, 8 insertions(+), 8 deletions(-)
    503:  Some shell scripts are not formatted. Please run 'make format_shell_scripts' to format and update them.
    504:  make: *** [Makefile:77: format_shell_scripts] Error 1
    505:  ##[error]Process completed with exit code 2.
    506:  ##[group]Run docker system prune -af
    

    Copy link

    qodo-merge-pro bot commented Jun 6, 2025

    CI Feedback 🧐

    A test triggered by this PR failed. Here is an AI-generated analysis of the failure:

    Action: Test Selenium Grid on Docker / Test Docker Selenium (test_video_standalone, false, true, false, ubuntu-24.04)

    Failed stage: Format and lint scripts [❌]

    Failed test name: format_shell_scripts

    Failure summary:

    The action failed because shell scripts in the repository are not properly formatted. The error
    occurred during the execution of the make format_shell_scripts target (Makefile:77). Specifically,
    the Video/video.sh file has formatting issues with 8 insertions and 8 deletions needed. The action
    is expecting all shell scripts to be formatted according to the project's standards.

    Relevant error logs:
    1:  ##[group]Operating System
    2:  Ubuntu
    ...
    
    148:  �[36;1m�[0m
    149:  �[36;1m  sudo rm -rf /opt/ghc || true�[0m
    150:  �[36;1m  sudo rm -rf /usr/local/.ghcup || true�[0m
    151:  �[36;1m  �[0m
    152:  �[36;1m  AFTER=$(getAvailableSpace)�[0m
    153:  �[36;1m  SAVED=$((AFTER-BEFORE))�[0m
    154:  �[36;1m  printSavedSpace $SAVED "Haskell runtime"�[0m
    155:  �[36;1mfi�[0m
    156:  �[36;1m�[0m
    157:  �[36;1m# Option: Remove large packages�[0m
    158:  �[36;1m# REF: https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh�[0m
    159:  �[36;1m�[0m
    160:  �[36;1mif [[ false == 'true' ]]; then�[0m
    161:  �[36;1m  BEFORE=$(getAvailableSpace)�[0m
    162:  �[36;1m  �[0m
    163:  �[36;1m  sudo apt-get remove -y '^aspnetcore-.*' || echo "::warning::The command [sudo apt-get remove -y '^aspnetcore-.*'] failed to complete successfully. Proceeding..."�[0m
    164:  �[36;1m  sudo apt-get remove -y '^dotnet-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^dotnet-.*' --fix-missing] failed to complete successfully. Proceeding..."�[0m
    165:  �[36;1m  sudo apt-get remove -y '^llvm-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^llvm-.*' --fix-missing] failed to complete successfully. Proceeding..."�[0m
    166:  �[36;1m  sudo apt-get remove -y 'php.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y 'php.*' --fix-missing] failed to complete successfully. Proceeding..."�[0m
    167:  �[36;1m  sudo apt-get remove -y '^mongodb-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^mongodb-.*' --fix-missing] failed to complete successfully. Proceeding..."�[0m
    168:  �[36;1m  sudo apt-get remove -y '^mysql-.*' --fix-missing || echo "::warning::The command [sudo apt-get remove -y '^mysql-.*' --fix-missing] failed to complete successfully. Proceeding..."�[0m
    169:  �[36;1m  sudo apt-get remove -y azure-cli google-chrome-stable firefox powershell mono-devel libgl1-mesa-dri --fix-missing || echo "::warning::The command [sudo apt-get remove -y azure-cli google-chrome-stable firefox powershell mono-devel libgl1-mesa-dri --fix-missing] failed to complete successfully. Proceeding..."�[0m
    170:  �[36;1m  sudo apt-get remove -y google-cloud-sdk --fix-missing || echo "::debug::The command [sudo apt-get remove -y google-cloud-sdk --fix-missing] failed to complete successfully. Proceeding..."�[0m
    171:  �[36;1m  sudo apt-get remove -y google-cloud-cli --fix-missing || echo "::debug::The command [sudo apt-get remove -y google-cloud-cli --fix-missing] failed to complete successfully. Proceeding..."�[0m
    172:  �[36;1m  sudo apt-get autoremove -y || echo "::warning::The command [sudo apt-get autoremove -y] failed to complete successfully. Proceeding..."�[0m
    173:  �[36;1m  sudo apt-get clean || echo "::warning::The command [sudo apt-get clean] failed to complete successfully. Proceeding..."�[0m
    174:  �[36;1m�[0m
    ...
    
    489:  ffmpeg -hide_banner -loglevel warning -threads ${SE_FFMPEG_THREADS:-1} -thread_queue_size 512 \
    490:  -        -probesize 32M -analyzeduration 0 -y -f x11grab -video_size ${VIDEO_SIZE} -r ${FRAME_RATE} \
    491:  -        -i ${DISPLAY} ${SE_AUDIO_SOURCE} -codec:v ${CODEC} ${PRESET:-"-preset veryfast"} \
    492:  -        -tune zerolatency -crf ${SE_VIDEO_CRF:-28} -maxrate ${SE_VIDEO_MAXRATE:-1000k} -bufsize ${SE_VIDEO_BUFSIZE:-2000k} \
    493:  -        -pix_fmt yuv420p -movflags +faststart "$video_file" &
    494:  +          -probesize 32M -analyzeduration 0 -y -f x11grab -video_size ${VIDEO_SIZE} -r ${FRAME_RATE} \
    495:  +          -i ${DISPLAY} ${SE_AUDIO_SOURCE} -codec:v ${CODEC} ${PRESET:-"-preset veryfast"} \
    496:  +          -tune zerolatency -crf ${SE_VIDEO_CRF:-28} -maxrate ${SE_VIDEO_MAXRATE:-1000k} -bufsize ${SE_VIDEO_BUFSIZE:-2000k} \
    497:  +          -pix_fmt yuv420p -movflags +faststart "$video_file" &
    498:  FFMPEG_PID=$!
    499:  if ps -p $FFMPEG_PID >/dev/null; then
    500:  recording_started="true"
    501:  Video/video.sh | 16 ++++++++--------
    502:  1 file changed, 8 insertions(+), 8 deletions(-)
    503:  Some shell scripts are not formatted. Please run 'make format_shell_scripts' to format and update them.
    504:  make: *** [Makefile:77: format_shell_scripts] Error 1
    505:  ##[error]Process completed with exit code 2.
    506:  ##[group]Run docker system prune -af
    

    Copy link

    qodo-merge-pro bot commented Jun 6, 2025

    PR Code Suggestions ✨

    Explore these optional code suggestions:

    CategorySuggestion                                                                                                                                    Impact
    General
    Use consistent sleep interval

    The random sleep duration could cause unpredictable behavior. Using a fixed
    interval would provide more consistent and predictable polling behavior, which
    is important for system stability and resource usage.

    Video/video.sh [292]

     +      echo "$(date -u +"${ts_format}") [${process_name}] - No session is created yet, waiting for a new session to start"
    -+      sleep $((RANDOM % 16 + poll_interval))
    ++      sleep ${poll_interval}

    [To ensure code accuracy, apply this suggestion manually]

    Suggestion importance[1-10]: 4

    __

    Why: The suggestion promotes predictable polling behavior, but the random component might be intentionally added to prevent synchronized polling from multiple instances. This is a minor behavioral optimization.

    Low
    Increase video encoding bitrates

    The -maxrate and -bufsize parameters are set too low for high-resolution videos,
    which could cause quality degradation. Increase these values or make them
    proportional to the video resolution to maintain better quality.

    Video/video.sh [235]

    -+  -tune zerolatency -crf ${SE_VIDEO_CRF:-28} -maxrate ${SE_VIDEO_MAXRATE:-1000k} -bufsize ${SE_VIDEO_BUFSIZE:-2000k} \
    ++  -tune zerolatency -crf ${SE_VIDEO_CRF:-28} -maxrate ${SE_VIDEO_MAXRATE:-2000k} -bufsize ${SE_VIDEO_BUFSIZE:-4000k} \

    [To ensure code accuracy, apply this suggestion manually]

    Suggestion importance[1-10]: 3

    __

    Why: While higher bitrates could improve video quality, the current values are configurable via environment variables (SE_VIDEO_MAXRATE and SE_VIDEO_BUFSIZE). This is a minor optimization suggestion since users can already adjust these parameters as needed.

    Low
    • Update

    Signed-off-by: Viet Nguyen Duc <nguyenducviet4496@gmail.com>
    @VietND96 VietND96 merged commit 204f199 into trunk Jun 7, 2025
    28 checks passed
    @VietND96 VietND96 deleted the video-high-cpu branch June 7, 2025 04:37
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    [🐛 Bug]: High CPU usage with 4.31 compared to 4.30
    1 participant