Analyze_commit_in_parallel lock out: 5 days and none completed

[cdolfi]

The instance as been on for 5 days and none of the tasks analyze_commit_in_parallel have completed and it seems like it has locked out the facade workers. Either that or there is an additional facade issue

[MoralCode]

DB logs show

2025-07-14 16:51:37.560 UTC [2889229] ERROR:  time zone displacement out of range: "2011-08-29 06:50:23 +51800" at character 1421
2025-07-14 16:51:37.560 UTC [2889229] STATEMENT:  INSERT INTO augur_data.commits (repo_id, cmt_commit_hash, cmt_author_name, cmt_author_raw_email, cmt_author_email, cmt_author_date, cmt_committer_name, cmt_committer_raw_email, cmt_committer_email, cmt_committer_date, cmt_added, cmt_removed, cmt_whitespace, cmt_filename, cmt_date_attempted, cmt_committer_timestamp, cmt_author_timestamp, tool_source, tool_version, data_source) VALUES ([Personal info redacted]

Turns out the specific commit that failed here is KNOWN for having invalid timezone info
https://github.com/search?q=4cf94979c9f4d6683c9338d694d5eb3106a4e734&type=code

several repos cite it in their comments as an outlier

While im not 100% certain this is the cause, this error is spammed over and over again, so i suspect one of the analyze_commit_in_parallel workers encountered this old broken commit and has just been holding a lock on large swaths of the database and stopping anything else from working.

What would the ideal behavior be in augur when encountering these invalid timestamps? skip the commit? try to correct them?

[MoralCode]

Source code ref where im thinking a fix is needed: https://github.com/chaoss/augur/blob/main/augur/tasks/git/util/facade_worker/facade_worker/analyzecommit.py#L212-L213

[MoralCode]

per gemini "+51800 seconds is over 14 hours, and the valid range is -12 to +14 hours"

Options to handle this:

• throw the whole commit away (loses a small amount of data, also probably relatively hard to implement)
• attempt to fix the value (how? maybe just modulus the value until its within standard range and adjust the date accordingly? truncate it?)
• reset the value to something known (i.e. a specific timestamp thats clearly not real - i.e. 1/1 1900 or something long before computers or commits would ever have been datestamped to)
• something else

It seems like rewriting the value in place and continuing on is the easiest path to implement

[MoralCode]

possibly relatd: #3134

[MoralCode]

despite my eagerness to solve this (and subsequently finding logs that are better suited to other issues), it seems like the scope of this issue is that the facade tasks are hanging.

[cdolfi]

re opening as it is not known if this issue is fixed

[cdolfi]

https://github.com/rivosinc/binutils-gdb this repo and its many mirrors/forks seems to have something in it thats a big issue