Error: socket hang up issue

[ilyasdresden]

Hello,

We faced with issue: Error: socket hang up.

Here was our setup:
MQTT (mosquitto) ->IoT Agent UL (FIWARE) -> Orion (FIWARE)
Then we pre-registered 1000 devices and started to send message over MQTT every 10 seconds with values for all devises.
At some point it stopped processing incoming data and showed this message in the log of IoT Agent UL.
We did the same with 100 devices it works well.

What can be the reason for that?
Is there any solution for that?

Thank you.

[fgalan]

At some point it stopped processing incoming data and showed this message in the log of IoT Agent UL

Thus, trasnferring this issue to IOTA-UL repository

[ilyasdresden]

@fgalan the problem is between IoT Agent and Orion.
MQTT gets everything and transfers to IoT Agent.
But IoT Agent tried to open a socket to Orion and it is not working.
Could it be that Orion could not handle that?
What is the capability of Orion Context Broker? How many incoming requests it can handle?
Thank you!

[fgalan]

It seems to be some kind of problem related with scarcity of resources.

Are all the systems (MQTT broker, IOTA UL and Orion) running in the same system? What about the free file descriptor? Can you monitor how many free file descriptors are in the operating system before starting sending measures and how they progress (i.e. free file descriptors in the operating system and file descriptors consumed by IOTA UL and Orion) during the test?

(Not sure the exact command to do that, but I think lsof should provide that information)

[AlvaroVega]

It would be a good idea to test this issue with latest version of this agent, that includes mqtt improvements.

[vijapandey]

Which version you had used to resolve the issue???

[fgalan]

Which version you had used to resolve the issue???

Not sure what you mean... our recomendation is always to use the newest version at the moment of the test. In this case, the newest version for IOTA-UL (at the present moment) is 1.24.0

[fgalan]

Anyway, as I explain in my previous comment I think this is not actually a problem in the IOTA-UL software, but some problem in the environment where it runs due to scarcity of resources or connectivity problems.

[pratappulugoru]

@fgalan we are still seeing this issue even if POD's are running with good number of resources. Our memory and CPU Utilization of POD's is very less.
Can you please guide on what may be the issue.

[fgalan]

To help on this, we would need a deterministic way of reproducing the problem. If you could provide that information, we could have a look.

[pratappulugoru]

@fgalan its very simple, please find the fiware stack configuration below in EKS services
iotagent-node-lib - 2.18.0 version.
iotagen t-json 1.20.0 version.
Orion - 3.4.0 version.
EKS Pod System Configuration:
Orion memory : 0.5 Gi
Pubsub pod count : Min -1 and Max -10
IOT Agent pod count : 1
Simulator Configuration:
And then send the Telemetry messages using any simulator like Jmeter or python program
Number of Devices registered - 840
Msgs/min - 1900
Duration - 2 hrs

With the above configuration, we can see the same behaviour

[pratappulugoru]

@fgalan any update on this. I have recorded the same issue in Orion also. Please see the below link
telefonicaid/fiware-orion#4429
Any help is highly appreciated

[pratappulugoru]

@fgalan as requested , I have attached all our deployment configuration files, Request you to go thorough and let me know if any changes required to solve Socket errors issue .
iot-agent.txt
mongodb-mask.txt
orion-qa-masked.txt