Some jobs launched through the map_function endpoint fail

[JavierGOrdonnez]

I called jc = functions_api_instance.map_function(function_uid, samples) and successfully got a RegisteredJobCollection back. When later retrieving the job results, some of them (about 10-20%, depending on the run) had failed.

This was in sim4life-staging.cloud. The two failed jobs in the screenshot above have ids:

• uid: ba4b6cf3-429f-4f48-81c8-96cbfbb574db ; projectJobId: 90a8c89e-4217-11f0-83a8-0242ac17ee66
• uid: cd5ce4b3-3f91-40fc-a5f6-fe1b80874745 ; projectJobId: aad043f0-4217-11f0-870e-0242ac17ee65

Thanks for looking into it.

[JavierGOrdonnez]

@sanderegg

Update after release of various fixes & patches (set to run at 8-9pm Tuesday 10th June).
The new batch had some failures (and some are still running).

Deployment: sim4life-staging.cloud
JobCollection UID: a93ab3a5-be49-4352-9cc7-57448f85e31f
UID of some failed jobs:

• 87d7d7a7-96cf-44d5-9ab7-c15eab28cb14
• 1999d57a-1f46-40c9-bbb2-416e492faeea
• 5c823bba-a870-4650-a518-23af2f05ff61
• 7d90093d-60b9-4667-b6c3-98e1a96a06a1

[bisgaard-itis]

We just investigated this together with @JavierGOrdonnez.
We were not able to reproduce the error. What we observed is that if the map endpoint in the api-server returns 200, then all jobs succeed. We also tried running this as two users at the same time and it worked fine. What does fail sometime is the map endpoint in the api-server. Here are the two reasons we observed for that happening:

• The 60s timeout in the api-server when waiting to create a job often causes issues. This is for sure an issue for studies with large amounts of data (see https://github.com/ITISFoundation/osparc-simcore/blob/85a2e9d08db780e3a3a840f0f63cb31e4aed3bc0/services/api-server/src/simcore_service_api_server/services_http/webserver.py#L225).
• Sometimes the rpc endpoints timeout (see https://monitoring.osparc-staging.io/graylog/messages/graylog_250/b4041241-46c6-11f0-ab1d-0242c0a80103)

[wvangeit]

So i would suggest to first focus on this: #1904

And see if we still get failed jobs afterwards (without the map returning an error directly).

[bisgaard-itis]

Closing this because we could not reproduce (see #1898 (comment)) and opening #1906 to fix timeout issues.