Connector deletes every printer and complains about Too many files open

[narvikmunicipality]

Hi

We're running a Debian Jessie installation with the cloud-print-connector, built using "GOPATH=/tmp/gcpbuild /usr/bin/go get github.com/google/cloud-print-connector/..." (Built March 7) and have been experiencing issues with it where it deletes every printer and then synchronizes them again, which in turn causes the sharing settings that was set on each printer to disappear.

From what we're seeing in the log it seems the connector gets caught in a loop fetching the same jobs over and over again, and when the limit for open files is exhausted it prints error messages like this:
E [16/Mar/2017:08:10:32 +0100] [Job REMOVED-GUID-JOB-13] Failed to create a temporary file: open /tmp/cloud-print-connector-395094484: too many open files

And as far I can tell on the issue with why the printers get deleted is because when the connector tries to do a synchronization, it too fails with "Too many files"-error when trying to contact the CUPS server, like this:
E [16/Mar/2017:08:11:36 +0100] [Printer Printer38] Failed to call cupsGetPPD3(): 1282 Too many open files
I [16/Mar/2017:08:11:37 +0100] [Printer Printer31 REMOVED-GUID-31] Deleted from the cloud

Here's the full log file with the messages the connector printed when deleting the printers.

[jay0lee]

Seems like a few issues here:

1. Because of the nature of cloudprint, we may see a job more than once between fetch calls. We should be checking if we're already working on a job after we see it in fetch so we don't do double duty.
2. We're current retrying EVERY HTTP status code other than 200 - Success. That's bad, we should only be retrying on 500 - 599 HTTP responses.
3. We'll keep retrying for up to 15 minutes. That's also probably bad. 5 minutes or even just 1 minute are probably more sane and it's unlikely we'll succeed very often in the 5 - 15 minute delay period.

I'm working on a patch for 2 which should prevent these issues by giving up much quicker on a 404 error. We should consider 1 and 3 also at some point but I suspect just 2 will fix this.

[bblietz]

Also see issue #343