DLPX-76203 Remove the NFS connection limit

[don-brady]

Context/Problem:

NFS connections are currently limited to a maximum of 1,340 connections. This arbitrary limit is calculated based on the maximum number of NFSD threads (currently 64). Note that on illumos, the maximum number of connections was unlimited by default. With the use of dNFS, we can bump up against this limit which subsequently can have a negative impact on NFS performance.

Solution:

Ideally we should not have to rely on increasing the number of NFSD threads in order to allow for more connections. So instead, we can set the connection limit to be independent of the NFS thread limit by using the NFSD tunable max_connections.

Testing

ab-pre-push: http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-orchestrator-pre-push/5772/

Checked the image from the ab pre push to confirm that max_connections is set whenever the nfs-server is starts up.

Implementation:

Uses an override to the nfs-server service to set the NFSD procfs variable for max_connections.
Note there is no nfs configuration for max_connections so the above is the best option for keeping it set persistently.

[sebroy]

Another thing to note on the testing side is that the performance team was able to easily demonstrate how hitting this connection limit would increase client-side I/O latencies due to the NFS client having to keep re-trying to establish connections in the context of trying to issue I/Os to the server.