Sync clock entries fails

[trentfisher]

When I try to sync my clock time (org-jira-update-worklogs-from-org-clocks) of some tasks I get this:

About to sync worklog for issue: FOOBAR-49 in file: FOOBAR
[error] request--curl-sync: semaphore never called [28 times]
start-process: Creating pipe: Too many open files
End of buffer [4 times]

It appears that at least some of the clock entries got put into Jira, but the 🆔 field was not added to my org file. If I do the sync again, it will eventually work, but I end up with duplicate entries (both in Jira and in my org file).

My guess is that the problem is the number of clock entries in this particular task, which are numerous.

Has anybody else seen this?

[trentfisher]

FYI, This is on Emacs 28.2 on Windows, and I am using the latest version of org-jira direct from GitHub (though older versions had the same problem)

I am adding some debug output to see if I can narrow down where things are going wrong.... but I don't really know much Emacs Lisp.

[ahungry]

How many clock entries are you trying to sync at once?

I searched the error message and found some interesting take-aways:

• https://emacs.stackexchange.com/questions/12842/creating-pipe-too-many-open-files-error-when-running-org-publish-on-large-pro
• Synchronous request deadlocks tkf/emacs-request#132
• https://www.reddit.com/r/emacs/comments/kgv4fj/problems_in_org_trello_error_requestcurlsync/

I'm guessing the underlying library that calls curl for jiralib is (possibly) making one temporary/intermediate file for each request. Could the sum of your open files + individual work log entries be greater than 512? (the first link mentions that as a hard limit on windows systems). Since all org-jira calls try to happen asynchronously, these could all be running at once.

If that's not the case - do either of the following links (link 2 or link 3) offer any useful info?

It'd be helpful as well if you had a way to test on GNU/Linux, so we could isolate this to a "just Windows" problem or not.

[trentfisher]

I am pretty sure that the number of worklogs is the problem, the task I am trying to update has 330 clock entries 56 of which have not been synchronized to Jira yet. But the Jira task itself has 502 clock entries (other people log time on this task). This is far bigger than any other task, and this seems to be the only one with the problem.

I did add some debug output and what I saw was that those 56 worklogs get put into Jira, and then it proceeds to update the already-sync'ed worklogs, and gets through 69 of them, and then the "[error] request--curl-sync: semaphore never called" messages are issued for each worklog update (28 of them) and then the next attempted update results in "start-process: Creating pipe: Too many open files" That adds up to 154.

I will investigate the links you sent, but I wanted to post what I have so far.

[ahungry]

I can confirm this package will try to run one HTTP call per worklog entry (as opposed to any sort of batch operation on all of them under a given ticket).

If you have that many worklog entries, it is likely the underlying issue. The code in org-jira-update-worklogs-from-org-clocks seems to iterate over one at a time, and (supposedly) call synchronously vs async, as it passes a nil for the callback function which controls this behavior - however I suspect it's still something being left behind in the request.el library.

Maybe a good test would be to write a generic wrapper around a request.el call to an endpoint, and ensure it loops at least 600 times - if the test succeeds, we can rule it out as the problem point - if not, we know the culprit (for all the good it'd do).

If Jira has a batch time log endpoint, and you can link some documentation for it, we might be able to enhance the time log handling to proceed in batches vs one item at a time, greatly reducing the calls.