inputs.processes fails with "no such process"

[noidi]

Bug report

Our Telegraf installations occasionally produce error messages like the following:

E! ERROR in input [inputs.processes]: read /proc/16314/stat: no such process

The error is triggered when a process is terminated while readProcFile (plugins/inputs/system/processes.go) is reading the process's /proc/<pid>/stat file.

System info:

All versions including the current master.

Steps to reproduce:

The bug is triggered by a race condition so it's quite hard to reproduce in Telegraf (we get 10-20 errors per day from 500+ hosts), but here's a little test program that uses a copy of readProcFile to demonstrate the issue.

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "os"
    "os/exec"
)

type reproductionType int

// Copied from plugins/inputs/system/processes.go
func readProcFile(filename string) ([]byte, error) {
    data, err := ioutil.ReadFile(filename)
    if err != nil {
        if os.IsNotExist(err) {
            return nil, nil
        }

        return nil, err
    }

    return data, nil
}

func main() {
    // Start cat with an open stdin pipe to keep the process running.
    cat := exec.Command("cat")
    catStdin, err := cat.StdinPipe()
    if err != nil {
        log.Fatal(err)
    }

    err = cat.Start()
    if err != nil {
        log.Fatal(err)
    }

    // In the background, close cat's stdin to make it terminate and wait for
    // the process to exit.
    isOpen := true
    go func() {
        catStdin.Close()
        cat.Wait()
        isOpen = false
    }()

    // As long as cat hasn't been terminated, keep rereading its stat file
    // from /proc.
    for isOpen {
        _, err = readProcFile(fmt.Sprintf("/proc/%d/stat", cat.Process.Pid))
        if err != nil {
            log.Fatal(err)
        }
    }
}

Running the test program multiple times tends to cause at least one failure:

$ for x in $(seq 10); do echo $x; ./proctest; done
1
2
3
4
5
2017/05/16 14:00:45 read /proc/22710/stat: no such process
6
2017/05/16 14:00:45 read /proc/22716/stat: no such process
7
8
9
10
2017/05/16 14:00:45 read /proc/22740/stat: no such process

Expected behavior:

Telegraf already checks for process termination between listing the /proc/*/stat files and reading them. Process termination between open() and read() should be handled the same way (i.e. without errors).

Actual behavior:

E! ERROR in input [inputs.processes]: read /proc/16314/stat: no such process