[Bug]: LD_PRELOAD cannot be preloaded

[roablep]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: milvusdb/milvus:v2.4.17
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): default(rocksmq)
- SDK version(e.g. pymilvus v2.0.0rc2): N/A
- OS(Ubuntu or CentOS): Ubuntu 24.04.1 LTS
- CPU/Memory: 
CPU(s):                               2
On-line CPU(s) list:                  0,1
Model name:                           AMD EPYC Processor
BIOS Model name:                      NotSpecified  CPU @ 2.0GHz
NUMA node0 CPU(s):                    0,1

               total        used        free      shared  buff/cache   available
Mem:           1.9Gi       1.6Gi       171Mi       6.0Mi       258Mi       263Mi

- GPU: 
- Others:

Current Behavior

ERROR: ld.so: object '/milvus/lib/' from LD_PRELOAD cannot be preloaded (cannot read file data): ignored.

Expected Behavior

docker compose up launches the container

Steps To Reproduce

Deploy Milvus using the provided docker-compose.yml.
Attempt to start the container.
Observe the container fails with exit code 134 and logs the above error.

Milvus Log

root@docker-ce-ubuntu-2gb-ash-1:~# docker inspect 
[
    {
        "Id": "sha256:4683fdeb1c31ef39c80a542a268f6bbe8dfdd1805e630362f6d98bb73de11a50",
        "RepoTags": [
            "milvusdb/milvus:v2.4.17"
        ],
        "RepoDigests": [
            "milvusdb/milvus@sha256:def1f0a3f90cd67273afba833ae68ed5cdadf6bb7a128b694b662323f74940b8"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2024-11-22T11:41:12.232359318Z",
        "DockerVersion": "24.0.5",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/milvus/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LD_LIBRARY_PATH=/milvus/lib::/usr/lib",
                "LD_PRELOAD=/milvus/lib/:/milvus/lib/libjemalloc.so",
                "MALLOC_CONF=background_thread:true"
            ],
            "Cmd": null,
            "Image": "sha256:41136788db4d6b90332900d5b4a934f9f8983c2167d8266ca85bb2c19764ee75",
            "Volumes": null,
            "WorkingDir": "/milvus/",
            "Entrypoint": [
                "/tini",
                "--"
            ],
            "OnBuild": null,
            "Labels": {
                "org.opencontainers.image.ref.name": "ubuntu",
                "org.opencontainers.image.version": "22.04"
            }
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 1664829469,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/15b745a5c0deecf32da0bb3fd3eca9dc06bd66d1b0bfd2c0ef87b3f9a2add0b5/diff:/var/lib/docker/overlay2/958654eac65bbe0d64e4ee459537372250b5b8ce39443fd557e520836d5ed501/diff:/var/lib/docker/overlay2/76ca9924f4f61d739fea0f539ef4cdd4fe74b39b29e7b0afc5a3334c935bb6bf/diff:/var/lib/docker/overlay2/7dc2a362c36815fb2ac618435b5efe6b22c6cf5bf9443ec17d5571e1aece25e7/diff:/var/lib/docker/overlay2/98ed5484c9b5010a060a966fd360f4e21d2bd00171e9ac2eaadf12f4f8805b01/diff",
                "MergedDir": "/var/lib/docker/overlay2/cf42bb2890955af49328a7d5cfe7196438f725b6213a9f08f096d510d5245a08/merged",
                "UpperDir": "/var/lib/docker/overlay2/cf42bb2890955af49328a7d5cfe7196438f725b6213a9f08f096d510d5245a08/diff",
                "WorkDir": "/var/lib/docker/overlay2/cf42bb2890955af49328a7d5cfe7196438f725b6213a9f08f096d510d5245a08/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:0b9c994b0484c0bc61f9de7c28a58745a504704254c5e8ed12349ebee3393a66",
                "sha256:c75ec3f1b42dcac8b1d9cd2e4e5b26eb29414a969ac4866a115212350bddfeb1",
                "sha256:8054a3c041e90a8422307b12c174ac445ea7b035ef08f8de8f0de4ac51d670c8",
                "sha256:6f9cc92b5334d3a11d924a1433b0725cba89a8f47e63f68475beb0a107a56b81",
                "sha256:60f673b3f11d932f93a8477f87214d52bb1aa7af7edf17f445025bab13406942",
                "sha256:e594bf1dbba9fb458d4b0b4dbac86f205866981a2465e9d105cea404cc8bef6a"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

Anything else?

I expect this has to do with the volume mapping, but I don't see anything inherently wrong with it.

  standalone:
    container_name: milvus-standalone
    networks:
      - milvus_net
    image: milvusdb/milvus:v2.4.17
    command: ["milvus", "run", "standalone"]
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - /home/<my_user>/volumes/milvus:/var/lib/milvus
    ports:
      - "19530:19530"
      - "9091:9091"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    depends_on:
      - "etcd"
      - "minio"

[LoveEachDay]

@roablep ERROR: ld.so: object '/milvus/lib/' from LD_PRELOAD cannot be preloaded (cannot read file data): ignored. won't block the startup.

Could you use lscpu to check the cpu flags? Read more from https://milvus.io/docs/operational_faq.md#How-do-I-know-if-my-CPU-supports-Milvus

[yanliang567]

/assign @roablep
/unassign

[roablep]

@LoveEachDay here you go:

root@docker-ce-ubuntu-2gb-ash-1:/home/ayla# lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          40 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   2
  On-line CPU(s) list:    0,1
Vendor ID:                AuthenticAMD
  BIOS Vendor ID:         QEMU
  Model name:             AMD EPYC Processor
    BIOS Model name:      NotSpecified  CPU @ 2.0GHz
    BIOS CPU family:      1
    CPU family:           23
    Model:                49
    Thread(s) per core:   1
    Core(s) per socket:   2
    Socket(s):            1
    Stepping:             0
    BogoMIPS:             4890.81
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_
                          freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perf
                          ctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr wbnoinvd arat umip rdpid arch_capabiliti
                          es
Virtualization features:
  Hypervisor vendor:      KVM
  Virtualization type:    full
Caches (sum of all):
  L1d:                    64 KiB (2 instances)
  L1i:                    64 KiB (2 instances)
  L2:                     1 MiB (2 instances)
  L3:                     16 MiB (1 instance)
NUMA:
  NUMA node(s):           1
  NUMA node0 CPU(s):      0,1
Vulnerabilities:
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Mitigation; untrained return thunk; SMT disabled
  Spec rstack overflow:   Vulnerable: Safe RET, no microcode
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Retpolines; IBPB conditional; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Not affected
  Tsx async abort:        Not affected

[roablep]

I see sse4_2, avx, and avx2

Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr wbnoinvd arat umip rdpid arch_capabilities

[roablep]

@LoveEachDay @yanliang567 Added requested info. Thanks!

[Mhsh]

I am getting same error while using milvus. I started evaluating hybrid search option of chroma db today.
I did below steps.

• 1 Ran docker pull milvusdb/milvus
• 2 Started container normally without any additional settings via docker dashboard


_Got below error in the logs and process exited_


2024-12-18 16:46:44 ERROR: ld.so: object '/milvus/lib/' from LD_PRELOAD cannot be preloaded (cannot read file data): ignored.
2024-12-18 16:46:44 tini (tini version 0.19.0 - git.de40ad0)
2024-12-18 16:46:44 Usage: tini [OPTIONS] PROGRAM -- [ARGS] | --version
2024-12-18 16:46:44 
2024-12-18 16:46:44 Execute a program under the supervision of a valid init process (tini)
2024-12-18 16:46:44 
2024-12-18 16:46:44 Command line options:
2024-12-18 16:46:44 
2024-12-18 16:46:44   --version: Show version and exit.
2024-12-18 16:46:44   -h: Show this help message and exit.
2024-12-18 16:46:44   -s: Register as a process subreaper (requires Linux >= 3.4).
2024-12-18 16:46:44   -p SIGNAL: Trigger SIGNAL when parent dies, e.g. "-p SIGKILL".
2024-12-18 16:46:44   -v: Generate more verbose output. Repeat up to 3 times.
2024-12-18 16:46:44   -w: Print a warning when processes are getting reaped.
2024-12-18 16:46:44   -g: Send signals to the child's process group.
2024-12-18 16:46:44   -e EXIT_CODE: Remap EXIT_CODE (from 0 to 255) to 0.
2024-12-18 16:46:44   -l: Show license and exit.
2024-12-18 16:46:44 
2024-12-18 16:46:44 Environment variables:
2024-12-18 16:46:44 
2024-12-18 16:46:44   TINI_SUBREAPER: Register as a process subreaper (requires Linux >= 3.4).
2024-12-18 16:46:44   TINI_VERBOSITY: Set the verbosity level (default: 1).
2024-12-18 16:46:44   TINI_KILL_PROCESS_GROUP: Send signals to the child's process group.

I am using macOS Sonama version 14.6.1 and below are machdep details

machdep.cpu.cores_per_package: 8
machdep.cpu.core_count: 8
machdep.cpu.logical_per_package: 8
machdep.cpu.thread_count: 8
machdep.cpu.brand_string: Apple M2
machdep.user_idle_level: 0
machdep.wake_abstime: 45908976192291
machdep.time_since_reset: 222310216431
machdep.wake_conttime: 235986986049656
machdep.deferred_ipi_timeout: 64000
machdep.virtual_address_size: 47
machdep.report_phy_read_delay: 0
machdep.report_phy_write_delay: 0
machdep.trace_phy_read_delay: 0
machdep.trace_phy_write_delay: 0
machdep.phy_read_delay_panic: 0
machdep.phy_write_delay_panic: 0
machdep.ptrauth_enabled: 1

[roablep]

@LoveEachDay @yanliang567 - Can you advise? Thank you!

[xiaofan-luan]

@LoveEachDay @yanliang567 - Can you advise? Thank you!

can you offer some log?
is there any fatal, error, warn message on your log?

Could you please refer this doc to export the whole Milvus logs for investigation?
For Milvus installed with docker-compose, you can use docker-compose logs > milvus.log to export the logs.

[xiaofan-luan]

I guess you may need more memory.
The minimal deployment we required is 8GB.
If you only have 2GB memory, maybe you can only serve 100K vectors

[vepicurean]

The same ERROR: ld.so: object '/milvus/lib/' from LD_PRELOAD cannot be preloaded (cannot read file data): ignored. when starting the container on Linux VM with lots of RAM ( over 8GB) on MacOS using Podman.