Move to self hosted runner

[yarikoptic]

running on github CI shows to be unreliable for testing connections (recently at least). I think the better way would be to move to some other network per https://upptime.js.org/docs/configuration/#self-hosted-runners howtos. @CodyCBakerPhD has experience with those. But ideally we do that on some DANDI centrally managed instance, not yet another use for an ad-hoc server (of mine). @kabilar @satra WDYT?

[CodyCBakerPhD]

some DANDI centrally managed instance

Such as MIT's engaging cluster?

[kabilar]

That sounds good. Either the Engaging Cluster or an EC2. I am going to disable the Uptime CI workflow for now. Too many false positives.

[satra]

before we switch since this has been a recent occurrence, has anybody checked why?

• did something change on the github side?
• is our upptime kept up to date with upstream?
• has anyone looked at other upptime installations? are they seeing similar issues?

with every new piece there is a new element to maintain and to monitor/update. with github we have somewhat limited that.

[yarikoptic]

some DANDI centrally managed instance

Such as MIT's engaging cluster?

nah... that's as centrally managed as "my" drogon server. I meant some cloud-y instance.

• did something change on the github side?

dunno but there were similarly intermittent issues here and there. https://www.githubstatus.com/history

• is our upptime kept up to date with upstream?

IIRC it is autoupdated via http://github.com/dandi/uptime//blob/HEAD/.github/workflows/updates.yml
etc

has anyone looked at other upptime installations? are they seeing similar issues?

interestingly I do not see any recent reports among https://github.com/upptime/upptime/issues?q=is%3Aissue%20state%3Aopen%20false%20positive

I am going to disable the Uptime CI workflow for now. Too many false positives.

@kabilar please do not disable generally... may be comment out your login for now and stop monitoring this repo... I will keep finger on pulse a bit longer and indeed if keeps unstable -- I will report

[kabilar]

I am going to disable the Uptime CI workflow for now. Too many false positives.

@kabilar please do not disable generally... may be comment out your login for now and stop monitoring this repo... I will keep finger on pulse a bit longer and indeed if keeps unstable -- I will report

Sounds good. I have re-enabled this workflow.

[asmacdo]

FWIW I briefly looked into increasing retries, but that doesnt appear to be exposed in the configuration.

[asmacdo]

Out of the box idea: if this is a gh action networking issue, could we detect the issue before running upptime? I'm thinking we ping google.com or something, and if that fails we cancel the job before upptime can open issues.

[CodyCBakerPhD]

FWIW I haven't observed many (if any) issues on GitHub runners specific to their own network connection, but I have seen plenty of website ping problems due to them blocking, restricting, or rate limiting the activity from GitHub runner IPs (which are easy to classify based on their public CIDR) so perhaps that might be the problem?

[asmacdo]

Good point, though IIUC our APIs don't have such restrictions so I have been assuming thats not the issue this time. Especially since they all seem to fail at the same time.

[yarikoptic]

yeah, observing no other similar reports I feel that it might relate to the number of checks we have enabled! So we might need to batch them up so a single run would not run through all of them.

[yarikoptic]

unstable performance on github CI persists -- inquired upstream

• Intermittent testing/high number of false positives upptime/upptime#1104

[yarikoptic]

FWIW, in the past days it seems nearly exclusively ontobee and purl... I think we should make those rare (once a week) if possible to reduce frequency of only some.