alt Hacker NewsHere is the thing, we have self-hosted runners running behind our firewall and even those go down for these kinds of outages. All the jobs just sit in the queue and are never sent to our self-hosted setup at all.
Replies
embedding-shape • yesterday at 8:06 PM
> we have self-hosted runners running behind our firewall and even those go down for these kinds of outages
Sounds like it's not as self-hosted as you think, or you have some other unnecessary dependency on something here? As long as GitHub isn't the sole place for your SCM, why would your self-hosted CI runners go down just because GitHub goes down?
1. Why isn't observability setup to monitor the queue depth and alert?
2. Why aren't the CI runners self-healing?