Skip to content

Under bandwidth pressure a j storage flash fails #118

@mangelajo

Description

@mangelajo

if j storage flash is uploading a file to the remote exporter, and there is bandwidth pressure, the upper jmp shell could end up closing the connection and the unix socket handed out to the lower processes (j storage/pytest/etc)

Image

This annotated log was generated with:
log_viewer.py

by usign

export GRPC_TRACE=all
export GRPC_VERBOSITY=info
jmp shell .... script.sh 2>grpc_jmp_shell.log

and a script.sh that did

j storage flash xxxxxx 2> grpc_logs.log

and then using the attached log viewer:
log_viewer.py

This stopped happened once we added:

grpcOptions:
  grpc.keepalive_timeout_ms: 120000

to the client, it turns out that we have a very low 5000 default, normally a TCP socket timeout is above 2 minutes, and also , the only reason why we were defaulting to grpc keepalive was to make sure that cloud load balancers would not drop our grpc connections due to innactivity.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    Status

    Done

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions