[Cypress Webkit Ci Run] Test stuck during Ci run

[TomdeHaan]

Current behavior

We are currently experimenting with cross browser testing within the CiCd pipeline. Our main goal is to run Cypress with Webkit and Chrome and run it in multiple viewports.

We run Cypress on multiple machines within Gitlab, adding the --parallel to connect to the Cypress Dashboard. We found out that with every run the Webkit tests got stuck and eventually timed out.

Example of one of the files:

Example of what it looks like within Gitlab:

Example of timeout log within gitlab:

We enabled the debugger, and when analysing it, we see that the cypress:server:util:process_profiler keeps repeating itself in the end

Desired behavior

All tests run and finish correctly.

Test code to reproduce

Cypress Version

10.11.0

Node version

v16.14.2

Operating System

cypress-image/browsers:latest-cypress-10.11.0 docker image

Debug Logs

cypress:server:browsers:webkit-automation received requestfinished { responseReceived: { requestId: '453', status: 200, headers: { 'content-type': 'image/png', date: 'Fri, 11 Nov 2022 09:25:31 GMT', 'access-control-allow-origin': '*', 'access-control-allow-credentials': 'true', 'x-content-type-options': 'nosniff', 'last-modified': 'Fri, 11 Nov 2022 09:25:31 GMT', etag: 'W/"ff8cc89-ea33c-5ed2e7897d023"', 'accept-ranges': 'bytes', 'cache-control': 'max-age=2629746, public', expires: 'Fri, 11 Nov 2022 09:30:30 GMT', 'x-xss-protection': '1; mode=block', via: '1.1 dfd84a17eaa88d79994b6524cab4931e.cloudfront.net (CloudFront)', age: '1639861', connection: 'keep-alive', 'keep-alive': 'timeout=5', 'transfer-encoding': 'chunked', 'strict-transport-security': 'max-age=15724800; includeSubDomains, max-age=63072000; includeSubdomains; preload', 'permissions-policy': 'geolocation=();midi=();notifications=();push=();sync-xhr=();microphone=();camera=();magnetometer=();gyroscope=();speaker=(self);vibrate=();fullscreen=(self);payment=();', 'x-cache': 'Hit from cloudfront', 'x-amz-cf-pop': 'DUB56-P1', 'x-amz-cf-id': 'EUKPC3vBU19zR_nsT4U8eoutO65EerEzUcAC-jC3NPL9lTDZ0NXajQ==' } } } +0ms
  cypress:server:browsers:webkit-automation received requestfinished { responseReceived: { requestId: '459', status: 200, headers: { 'access-control-allow-origin': 'https://www.ziggo.nl', date: 'Wed, 30 Nov 2022 08:56:32 GMT', pragma: 'no-cache', expires: 'Fri, 01 Jan 1990 00:00:00 GMT', 'cache-control': 'no-cache, no-store, must-revalidate', 'last-modified': 'Sun, 17 May 1998 03:00:00 GMT', 'x-content-type-options': 'nosniff', 'access-control-allow-credentials': 'true', 'content-type': 'image/gif', 'cross-origin-resource-policy': 'cross-origin', connection: 'keep-alive', 'keep-alive': 'timeout=5', 'transfer-encoding': 'chunked', server: 'Golfe2', 'alt-svc': 'h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="46, 43"' } } } +11ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51776 } +4s
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51824 } +933ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51836 } +82ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51830 } +26ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51846 } +0ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51844 } +18ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51838 } +48ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51826 } +16ms
  cypress:server:browsers:webkit-automation received request { browserPreRequest: { requestId: '463', method: 'POST', url: 'https://region1.google-analytics.com/g/collect?v=2&tid=G-TH972FDWFF&gtm=2oebs0&_p=776742999&cid=872464754.1669798568&ul=en-us&sr=1440x1024&_eu=AAg&_s=9&sid=1669798568&sct=1&seg=1&dl=https%3A%2F%2Fwww.ziggo.nl%2Finternet%2Finternet-only%3Futm_medium%3Dtesting%26utm_source%3Dtesting%26optimizely_disable%3Dtrue&dt=Alleen%20Internet%20Thuis%20%E2%80%93%20Internet%20Only%20Pakketten%20Vergelijken%20%7C%20Ziggo&en=click_on_button&ep.interaction_event=click.button&ep.element_text=ga%20verder&ep.element_link=none&ep.element_classes=z-basket__footer-btn%20btn%20btn-primary&_et=2', headers: { accept: '*/*', referer: 'https://www.ziggo.nl/', origin: 'https://www.ziggo.nl', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15', 'cache-control': 'max-age=0' }, resourceType: 'other', originalResourceType: 'beacon' } } +5s
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51850 } +255ms
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   83.81    │     68.01      │ 1097.35  │    887.91    │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    1.89    │     14.79      │   295    │    248.49    │     295     │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    188.23    │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.54      │  137.66  │    182.66    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1497, 1498'       │     0      │       0        │   3.48   │     3.47     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │    85.7    │     75.73      │ 1733.46  │   1391.79    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   81.64    │     69.52      │ 1072.17  │    908.38    │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    0.3     │     13.34      │  295.07  │    253.14    │   295.07    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    189.4     │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.48      │  137.66  │    177.66    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1520, 1521'       │     0      │       0        │   3.55   │     3.48     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │   81.94    │     76.35      │ 1708.42  │   1423.45    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   86.33    │     71.21      │ 1063.43  │    923.89    │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    0.3     │     12.16      │  295.23  │    256.97    │   295.23    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    190.36    │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.43      │  137.66  │    173.66    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1544, 1545'       │     0      │       0        │   3.35   │     3.47     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │   86.63    │     77.28      │ 1699.65  │   1448.56    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:socket-base socket-disconnecting ping timeout +37s
  cypress:server:socket-base socket-disconnect ping timeout +0ms
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   84.24    │     72.39      │ 1123.76  │    942.06    │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    1.55    │     11.27      │  268.29  │    257.91    │   295.23    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    191.16    │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.39      │  137.66  │    170.38    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1569, 1570'       │     0      │       0        │   3.33   │     3.46     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │   85.79    │     77.99      │ 1733.01  │   1472.27    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   85.64    │     73.49      │ 1108.19  │    955.9     │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    3.54    │     10.68      │  230.77  │    255.83    │   295.23    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    191.84    │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.36      │  137.66  │    167.66    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1592, 1593'       │     0      │       0        │   3.47   │     3.46     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │   89.18    │     78.85      │ 1680.05  │   1488.25    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s

Other

No response

[flotwig]

@TomdeHaan can you share the full DEBUG logs please? The tail end shared here doesn't contain much info about what could be causing Cypress to hang.

[modern-sapien]

I followed up with @TomdeHaan this AM to see what the status of logs were to help move this forward.

[modern-sapien]

@TomdeHaan could you or one of the team members try your current test suite on the binary below & if this issue isn't resolved also try on 10.7.0.

Our team believes that this issue is more than likely related to #23830.

https://cdn.cypress.io/beta/npm/10.8.1/linux-x64/10.8-minus-video-refactor-a8279f8104c46a046d37a71991aa0eccb329a202/cypress.tgz

Please let us know here any relevant outcomes.

[me4502]

Different setup to OP's, but we have this same issue (after tests complete, the debug logs spam about the profiler until it times out), and for us it started on 10.11.0. If we stay on 10.10.0 we're fine, but any version above this, including 12.0.1, seems to trigger this problem.

(To clarify different setup; I mean we're hitting this on Chrome, not WebKit, and are using Node 18. We did have the same issue on Node 16 when we initially tried the 10.11.0 upgrade prior to the Node 18 upgrade)

[TomdeHaan]

@modern-sapien. Sorry, only saw this message on and of working day. Small update form our side.

We will try your suggestion next working week, but we also did some investigation our selves:.

Currently we are running Cypress inside Gitlab, using a docker images that we create ourselves, using this image: https://hub.docker.com/layers/cypress/browsers/latest/images/sha256-44f4157bfb319fccf537b590841315c9f777937f02f930e9d295e5cbdeb02381?context=explore.

We also install the webkit browser from playwright as mentioned in the documents https://docs.cypress.io/guides/guides/launching-browsers#Browsers.

# install webkit for safari
RUN npx playwright install-deps
`
We tried to run the tests as simple as possible, so we disabled the --parallel, recording and group. But that also failed. And also tried to run with Cypress 12.0.2. (Which we also created a docker image for).
It looks like (to us) that the connection is lost between browser and Cypress, see this log:

This code keeps repeating until it times out. 
"getDelayMsForRetry: �[36m[Function: b]�[39m }, iteration: �[33m0�[39m } �[32m+24ms�[0m
  �[32;1mcypress:https-proxy �[0mreceived upstreamSocket callback for request { port: �[33m41443�[39m, hostname: �[32m'localhost'�[39m, err: �[90mundefined�[39m } �[32m+1ms�[0m
  �[36;1mcypress:server:util:socket_allowed �[0mallowing socket { localPort: �[33m48946�[39m } �[36m+23ms�[0m
  �[32;1mcypress:https-proxy �[0mGot first head bytes { url: �[32m'static.ads-twitter.com:443'�[39m, head: �[32m`\x16\x03\x01\x02�\x01\x00\x02�\x03\x035��$�\x16\x11��(��~C4\x120��h\x1C\x195+|S("��� q�,\x18���5�����v��~!r'K`�[39m } �[32m+15ms�[0m
  �[32;1mcypress:https-proxy �[0mMaking intercepted connection to 41443 �[32m+0ms�[0m
  �[32;1mcypress:network:connect �[0msuccessfully connected { opts: { port: �[33m41443�[39m, host: �[32m'localhost'�[39m, getDelayMsForRetry: �[36m[Function: b]�[39m }, iteration: �[33m0�[39m } �[32m+16ms�[0m
  �[32;1mcypress:https-proxy �[0mreceived upstreamSocket callback for request { port: �[33m41443�[39m, hostname: �[32m'localhost'�[39m, err: �[90mundefined�[39m } �[32m+1ms�[0m
  �[36;1mcypress:server:util:socket_allowed �[0mallowing socket { localPort: �[33m48948�[39m } �[36m+16ms�[0m
  �[32;1mcypress:https-proxy �[0mGot first head bytes { url: �[32m's.pinimg.com:443'�[39m, head: �[32m'\x16\x03\x01\x02�\x01\x00\x02�\x03\x03�,�������Q\x02�\x06����ݷ�z�C��M%��m� o���pQ_����1g��*t��R��'�[39m } �[32m+15ms�[0m
  �[32;1mcypress:https-proxy �[0mMaking intercepted connection to 41443 �[32m+0ms�[0m
  �[32;1mcypress:network:connect �[0msuccessfully connected { opts: { port: �[33m41443�[39m, host: �[32m'localhost'�[39m, getDelayMsForRetry: �[36m[Function: b]�[39m }, iteration: �[33m0�[39m } �[32m+16ms�[0m
  �[32;1mcypress:https-proxy �[0mreceived upstreamSocket callback for request { port: �[33m41443�[39m, hostname: �[32m'localhost'�[39m, err: �[90mundefined�[39m } �[32m+1ms�[0m
  �[36;1mcypress:server:util:socket_allowed �[0mallowing socket { localPort: �[33m48950�[39m } �[36m+16ms�[0m
  �[32;1mcypress:https-proxy �[0mGot first head bytes { url: �[32m'deploy.mopinion.com:443'�[39m, head: �[32m"\x16\x03\x01\x02�\x01\x00\x02�\x03\x03N�3�1=q�\x16RB��~�It\x18��s�EW�d9Ӏ�`� �WZ��\x16�]�C�Q|��`'�m��"�[39m } �[32m+20ms�[0m
  �[32;1mcypress:https-proxy �[0mMaking intercepted connection to 41443 �[32m+0ms�[0m
  �[32;1mcypress:network:connect �[0msuccessfully connected { opts: { port: �[33m41443�[39m, host: �[32m'localhost'�[39m, getDelayMsForRetry: �[36m[Function: b]�[39m }, iteration: �[33m0�[39m } �[32m+21ms�[0m
  �[32;1mcypress:https-proxy �[0mreceived upstreamSocket callback for request { port: �[33m41443�[39m, hostname: �[32m'localhost'�[39m, err: �[90mundefined�[39m } �[32m+1ms�[0m
  �[36;1mcypress:server:util:socket_allowed �[0mallowing socket { localPort: �[33m48952�[39m } �[36m+21ms�[0m
  �[32;1mcypress:https-proxy �[0mGot first head bytes { url: �[32m'www.google-analytics.com:443'�[39m, head: �[32m'\x16\x03\x01\x02�\x01\x00\x02�\x03\x03��d=�$�ʆ�\x19\x06���Ы3�q�ȕ\x16�c\x15d�\x07 �Q�%�\x12��~\x1E=��Z�\x7Fؓφ�������
�[31;1mJob's log exceeded limit of 4194304 bytes.�[0;m".

If you want to have more details, please contact us again. And we will update, after we tried your suggestion.

[TomdeHaan]

@modern-sapien After the weekend, the team created an image with the version you provided. Question from the team, the title from your version suggests that this is without Webkit while we want to test with Webkit ofcourse. So that set us a little question mark.
job-wehkit-12-12-2022.log

We first run the test without using the Cypress dashboard and the tests passed (no --group, --parallel --record).

Then we activated all that again, and the same timeout as before. We attached the log file to this comment.