-
Notifications
You must be signed in to change notification settings - Fork 13
Closed
Milestone
Description
if j storage flash is uploading a file to the remote exporter, and there is bandwidth pressure, the upper jmp shell could end up closing the connection and the unix socket handed out to the lower processes (j storage/pytest/etc)
This annotated log was generated with:
log_viewer.py
by usign
export GRPC_TRACE=all
export GRPC_VERBOSITY=info
jmp shell .... script.sh 2>grpc_jmp_shell.log
and a script.sh that did
j storage flash xxxxxx 2> grpc_logs.log
and then using the attached log viewer:
log_viewer.py
This stopped happened once we added:
grpcOptions:
grpc.keepalive_timeout_ms: 120000
to the client, it turns out that we have a very low 5000 default, normally a TCP socket timeout is above 2 minutes, and also , the only reason why we were defaulting to grpc keepalive was to make sure that cloud load balancers would not drop our grpc connections due to innactivity.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Status
Done