DLPX-75405 nfs_threads script should flush output and include timestamps

[don-brady]

Problem:

The /usr/bin/nfs_threads script on the Delphix engine does not currently flush its output so redirecting it to a file results in an empty file. It is useful for Support to be able to capture this output when troubleshooting NFS issues.

To be able to match up the output with other estat data, it would also be useful to have it include timestamps on each line.

Solution:

The following improvements were added to nfs_threads script

• Added timestamp to output
• Print header every 48 lines of output
• Added flush to print lines
• Accommodate an NFS server restart while script is running
• Added optional --interval argument to supply sampling interval
• Fixed threads used calculation (was sometimes going negative)
• Fixed underflow in sockets-enqueued

Testing

ab-pre-push:
http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-orchestrator-pre-push/5276/

Manually tested on dcol1
Manually tested to confirm that flush works (can now tail output from nohup output file)

[tonynguien]

Looks good. Thanks for adding the timestamp :)