Skip to content

Runtime Failures on TCP Stress Testing: (libuv) kqueue(): Too many open files and more #12666

Closed
@ehsanul

Description

@ehsanul

I'm on OSX 10.8.4, x86_64, latest Rust on master. And to head off any concerns about my file handle limits:

$ ulimit
unlimited

I ran into this when stress-testing my web sockets library. After just a couple hundred connections, some connections were being refused at random. The only thing I could see was some output with RUST_LOG=debug saying (libuv) kqueue(): Too many open files. I've come up with a smaller test case that brings this out with some spectacular errors:

use std::io::net::tcp::{TcpListener, TcpStream};
use std::io::net::ip::{Ipv4Addr, SocketAddr};
use std::io::{Acceptor, Listener};
use std::io::IoResult;

fn main() {
    let addr = SocketAddr { ip: Ipv4Addr(127, 0, 0, 1), port: 8989 };
    spawn(proc() {
        let mut acceptor = TcpListener::bind(addr).listen();
        loop {
            let mut stream = match acceptor.accept() {
                Ok(stream) => stream,
                Err(error) => {
                    debug!("accept failed: {:?}", error);
                    continue;
                }
            };
            println!("{:?}", stream.read_byte().unwrap());
        }
    });
    for _ in range(0, 1000) {
        spawn(proc() {
            match TcpStream::connect(addr) {
                Ok(stream) => {
                    let mut stream = stream;
                    stream.write([1]);
                    let mut buf = [0];
                    stream.read(buf);
                },
                Err(e) => debug!("{:?}", e)
            }
        });
    }
}

Fails less often with 100-200 connections. The errors are different every time I run this with RUST_LOG=debug.

Here's the end of one run:

(libuv) kqueue(): Too many open files
bug(54921,0x10881c000) malloc: *** error for object 0x7fca4b620f70: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
std::io::IoError{kind: OtherIoError, desc: "too many open files", detail: None}
task '<unnamed>' failed at 'assertion failed: `(left == right) && (right == left)` (left: `140506823655696`, right: `140506824839680`)', /Users/harish/rust/rust/src/librustuv/homing.rs:110
(libuv) kqueue(): Too many open files
std::io::IoError{kind: OtherIoError, desc: "too many open files", detail: None}
task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:199
Abort trap: 6

Another:

task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:199
task failed during unwinding (double-failure - total drag!)
rust must abort now. so sorry.
Segmentation fault: 11

Another:

task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:199
task failed during unwinding (double-failure - total drag!)
rust must abort now. so sorry.
Illegal instruction: 4

And lastly:

fatal runtime error: leaked managed memory (140518327703008 objects)
(libuv) kqueue(): Too many open files
task '<unnamed>' failed at 'didnt want an unwind during: going home', /Users/harish/rust/rust/src/librustuv/lib.rs:Abort trap: 6

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions