logoalt Hacker News

Incident with multple GitHub services

257 pointsby bwannasekyesterday at 4:21 PM121 commentsview on HN

Comments

embedding-shapeyesterday at 4:59 PM

I've been on a somewhat binge to move a bunch of stuff to self-hosting at home. Yesterday I finally completed my self-hosted Forgejo instance at home, together with Linux, Windows (via VM) and macOS (via Mac Mini) runners/workers for CI/CD, so everything finally lives in-house (literally), instead of all source code + Actions being on GitHub but the infrastructure actually living locally.

This is probably the first time I felt vindicated with my self-hosting move literally the day after I finished the migration, very pleasant feeling. Usually it takes a month or two before I get here.

show 11 replies
LorenDByesterday at 5:09 PM

https://mrshu.github.io/github-statuses/ says they are down to 88.15% uptime. Even when you consider uptime of individual components, their best is 99.78%, so two nines.

show 2 replies
hedayetyesterday at 10:29 PM

Is Github losing any significant business from all these outages?

Curious because for a long time we as an industry maintained that reliability and brand value are business critical; but seems like they are cared very little now a days.

Happy to be corrected about my perception too.

show 1 reply
dankobgdyesterday at 5:01 PM

Don't worry, status page says that it's 100% working - green color, all good. even though i can't access a static page

eshack94today at 4:51 AM

At this point, I feel like there should be a HN post whenever there ISN'T an issue with some GitHub service. Otherwise, it's business as usual...

cjonasyesterday at 4:50 PM

It would be wild if they dropped below the "two 9's" metric. I think they would need an additional ~16hr of outage in the 90 day rolling period.

show 2 replies
frereubuyesterday at 8:40 PM

We have pretty basic needs - git repos + actions - and a bit of downtime here and there doesn't really affect us too much because we're not constantly committing and deploying, but even we're looking around for alternatives now.

Also, looks like people might be pummelling the SourceHut servers looking for an alternative: https://sr.ht/ is down. (Edit: was down when I wrote that, back up now).

show 1 reply
agartneryesterday at 10:01 PM

There was another really bad incident today: https://www.githubstatus.com/incidents/zsg1lk7w13cf

> We have resolved a regression present when using merge queue with either squash merges or rebases. If you use merge queue in this configuration, some pull requests may have been merged incorrectly between 2026-04-23 16:05-20:43 UTC.

We had ~8 commits get entirely reverted on our default branch during this time. I've never seen a github incident quite this bad.

show 3 replies
x0rumantoday at 4:01 AM

We lost about a day of git history across several repos on Bitbucket a while back-not an outage, a data issue on their side. Local clones saved most of it, but issues and PRs from that window were just gone. That’s roughly why I started building gitbacker as a side project. Turns out the ‘back up the repo’ part is easy; the metadata is where it gets interesting.

Groxxyesterday at 9:35 PM

So... three incidents today, all of them ~1h or longer, and everything's green for the day with "no recorded downtime".

These don't really look any different than past incidents which have red bars on their respective days, except maybe that those tended to be several hours.

What do the green bars even mean? Are they changed to non-green retroactively if people complain enough or something? So far as I can tell, literally none of the previous green days have any incident shown in the mouse-over, but there are multiple for today only, so I kinda have to assume the mouse-overs are conveniently "forgotten" or all incidents become non-green and they just don't bother informing anyone on the same day. Either seems intentionally misleading.

delusionalyesterday at 9:00 PM

Once that 10x developer velocity from AI kicks in, I'm sure github stability improves. Did you know AI finally makes it economical to fix all the little bugs?

CamouflagedKiwiyesterday at 5:40 PM

I wondered. We'd seen for most of today that Actions were slow to trigger, I had at least one that was just missed, it felt like something was definitely off but the status was green all day until this.

BhavdeepSethitoday at 4:23 AM

I'd really like to read the Post Mortem doc for this. I'm still shocked how they allowed this to happen in production.

jonnonzyesterday at 7:19 PM

Well I suppose they are finding out if you lay off too many people the IP of how the system works goes out the door with them.

throwatdem12311yesterday at 5:08 PM

Seems like they just can’t deal with the absolute deluge of AI vomit being uploaded every day.

Good riddance I hope it completely destroys them.

show 1 reply
argeeyesterday at 4:47 PM

I moved to Gitlab a while ago. It's a whole new level of freedom not having to pay for self-hosted CI runners.

redwoodtoday at 10:08 AM

Guessing the sheer volume of pull requests related to AI code jitter is leading to instability of this Microsoft product.

deferredgrantyesterday at 10:23 PM

These outages are also a good test of how much local resilience teams actually built. My guess is most shops are much more dependent on GitHub than they like to admit.

bakiesyesterday at 5:25 PM

I definitely have better uptime hosting my own gitea instance. It's faster too. It's basically a knock off GitHub. Plus with privacy concerns, I'm just happier overall. Easy setup, all I did was deploy the helm chart.

jasoncartwrightyesterday at 5:23 PM

Just cancelled my GitHub Copilot Pro+ year subscription. Removal of Opus 4.6 stung, but the repeated continued downtime makes it unusable for me. Very disappointed.

No fuss instant refund of my unused subscription (£160) appreciated.

show 2 replies
fishgoesblubyesterday at 4:50 PM

At this point it'll be better to have alerts for when GitHub is online, rather than offline.

AnkerSkallebankyesterday at 4:51 PM

Some of my jobs are completing, some are failing. Seems to be random. Kind of wish they would just fail outright, instead of running for 10 minutes and then failing.

surya2006yesterday at 5:05 PM

even vercel also have more downtime nowadays

supakeenyesterday at 4:42 PM

I mean; this is the normal mode of operation for GitHub at this point.

show 1 reply
surya2006yesterday at 5:04 PM

what are the good alternatives available for github i find some alternative but as long as widely people use github i cant use other service right like i cant share my alternative to other developer and force him to use this for me. so i feel like i locked in even i want to move i can't

show 5 replies
tomjen3yesterday at 5:22 PM

At this point it should almost be news when it works.

ChrisArchitectyesterday at 4:58 PM

Multple 9s

buildbotyesterday at 5:08 PM

Anyone also seeing Active Directory/Entra issues?

chrisweeklytoday at 1:30 AM

mods: title tupo "multple"

josefritzishereyesterday at 4:51 PM

Microslop is destroying Github

nhhvhyyesterday at 7:00 PM

If the day ends in Y…

linhnsyesterday at 4:57 PM

Business as usual.

0xbadcafebeeyesterday at 4:57 PM

I am this > < close to just running Gogs or Forgejo on some Hetzner boxes, quit my job, charge people for access. Why aren't there like 10 startups doing this yet? Please? I want to give you my money. Just give me a git host that doesn't suck. (All the current ones suck)

show 2 replies
poplarsolyesterday at 7:20 PM

Azure webapp deploys are also trash right now. Microsoft needs to stop slathering h1b copilot slop and get basic things like Windows patches working.

shevy-javayesterday at 5:05 PM

Microsoft again.

I think it is time that Microsoft lets go of GitHub. They are handling it too poorly.

bmd1905today at 3:59 AM

[dead]

ossa-mayesterday at 4:55 PM

Seems like outages are increasingly more frequent nowadays. Obviously, this is not the best state of affairs, and developers should not be limited by services. In the meantime I've been experimenting with building third spaces for people to chill while they wait for the services they are dependent on to go back up.

The first one I've built is a little ASCII hangout for Claude @ https://clawdpenguin.com but threads like this make me want to build it for Github too.