Skip to content

"Too many open files" error on tiering in Column shard #8257

Open
@swalrus1

Description

@swalrus1

Column shard creates unlimited number of file descriptors to connect to S3 for reading evicted blobs. If number of opened file descriptors reach ulimit -n, either VERIFY fails or scans that touch evicted rows fail with error "cannot get blob: curlCode: 6, Couldn't resolve host name". On scenario test setup, one descriptor were opened for every blob evicted to S3.

Failed verify:

VERIFY failed (2024-08-26T08:33:20.336761Z): grpc failed accept4: Too many open files
  contrib/libs/grpc/src/core/lib/iomgr/tcp_server_posix.cc:349
  on_read(): requirement false failed
??+0 (0xACB90CD)
??+0 (0xACB2DF7)
??+0 (0x11AFF6F7)
??+0 (0x11A33481)
??+0 (0x11AF06D7)
??+0 (0x11AEE96B)
??+0 (0x11AFA953)
??+0 (0x11AE1168)
??+0 (0x11D6F6F0)
??+0 (0x1564C27E)
??+0 (0xBA3A809)
??+0 (0xBA3AAEA)
??+0 (0xACBD7DF)
??+0 (0x7FA649C36609)
clone+67 (0x7FA649B56353)

Reproduce by running tiering scenario test #8262 until eviction.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions