My schools entire assignment system is out today.
You’d be hard pressed to find an online service that isn’t associated with AWS in some way.
Sadly, there are some who don’t even know it, because they’re buying services from someone else that buys them from someone else that buys them from Amazon. So they’re currently wondering what the fuck is even going on, since they thought they weren’t using AWS.
Well those people are fucking idiots.
That’s not really fair, I think. Smaller organizations are especially dispositioned here. Think small businesses, charities, local municipal services, etc. Small IT budgets, low staff (if any) and just enough to pad out a subscription cost to a service provider that fits their needs.
AWS is an incredibly low cost solution, and it’s probably where most of these low cost services point themselves at when building platforms at scale. Not everyone can build and maintain a datacentre or home server for their every need.
This isn’t to say that there are definitely idiots who pad their resume by chanting a prayer to SaaS and boasting about having moved their company to the “cloud” via a cheap and unreliable AWS rehoster, before failing upwards though.
Fine, most of them are fucking idiots. Know where your infrastructure is people! Whois your IP.
Do you know what power stations feed your electrical circuit? Do you know which transformers are most critical to keep you in service? Do you know who manufactured them?
Do you know where your water comes from? Which facilities treat it? How it’s treated?
Can you name your senators, house rep, state senators, state rep, and local officials?
Everyone can’t know everything. I doubt you do, but I wouldn’t call you a fucking idiot for that. I would call you a fucking idiot for being such a small-minded asshole though. You’re not the smartest person in the room, and even if you were, you still wouldn’t understand how 99% of human technology works. We’re in this together and we lean on each other. That’s the beauty and curse of being human
Sounds like you’ve taken offense. Don’t know who your host is huh? If you work in IT doing websites you gotta know same basics. Sorry if my statements don’t align with your world view.
Think small businesses, charities, local municipal services, etc. Small IT budgets, low staff (if any) and just enough to pad out a subscription cost to a service provider that fits their needs.
Did we not read the same comment? Not every service that was taken down was a website, like OP’s school. Plenty of small businesses, charities or local municipal services don’t have IT staff, commissioned their sites (including initialisation) and don’t necessarily know that 𝑥 company they pay, pays Amazon. Or how to do a whois search. You clearly don’t have a firm grasp on the realities of the world, nor work in IT “doing websites”.
I’m pretty sure most of Azure (Microsoft), OCI (Oracle), and GCP (Google) have all been fine.
Bezos is a craven beast but I don’t see many companies above with CEOs that I’d feel comfortable babysitting my teenage daughter
The company I work for is an Azure shop. However, our provider for customer 2fa tokens uses AWS… So still in trouble.
Once Larry Ellison owns TikTok he’s going to be babysitting all the teenagers and a whole bunch of other people!
I really hope that super villain wannabe croaks out real soon
I don’t think his son is any better.
If daddy croaks now we will see how much is actually David’s doing vs Larry
Sure, but online services can certainly leverage multiple modules, from multiple companies, hosted in multiple places. So maybe your site mostly works fine, but a key aspect of it is broken.
from multiple companies
See the above post from the Azure shop … that uses AWS for 2FA tokens
You want to add multiple companies in parallel as alternates/failovers, not in serial where any one failure blocks the whole flow
But that would cost more money, that’s anticapitalist.
Yes, it’s much more expensive to have two providers. Both in terms of outright costs but even more so in terms of ongoing engineering/technical overhead.
The calculus is how much the expectation downtime is, versus that cost. It’s a reasonable calculation and TBH if outages are a few hours once every few years for most cases it’s acceptable.
OFC if your hospitals or emergency services depend on a cloud service, you happily fork over the extra money same as you do for any other insurance.
If there’s anything I know, it’s that “businesspeople” are never proactive.
Also allow things fail gracefully, independent of each other.
Lemmy seemed fine, Reddit did not.
Lemmy seemed fine
Federated, open source
Reddit did not
Centralized, corporate
That doesnt really have anything to do with this issue. Lemmy can absolutly be hosted in AWS via ECS (or EKS if you love Kubernetes). Hell, it could be hosted directly on EC2 if preferred.
Federation as a whole is more resilient because each operator can chose whatever hosting solution they prefer. But if your particular server happened to be hosted in AWS in the useast1 region; your shit was gonna be a bit busted.
Amen to that, good thing though. Got me to learn what Lemmy was. Apparently I’ve been under a rock.
Welcome to the fediverse!!
Walmart.com would likely work fine, as they are rabidly anti-Amazon, especially AWS. They don’t even want their SaaS vendors using AWS under the covers for them.
Can confirm, about 10 years ago, the company I worked for migrated to AWS, and I managed the transition. We planned everything meticulously so that there would be no downtime, and used it as excuse to fix a lot of tech debt. No one was supposed to even notice the cutover, and when we did it, I expected the only feedback to be that things seemed faster and were working as expected. A few hours later, we get a complaint from an Account Manager for Walmart that they can’t access the platform at all. There was a lot of confusion and back and forth, turns out their IT department had an allow list or something in the corporate DNS to not resolve to AWS owned IPs unless approved. We eventually got them to add our domain to their allowlist, but it seemed insane that they would spend the effort to implement and maintain that level of control.
Amazon and walmart are competitors
Totally, I understand that, but seemed to be an extreme measure they are inflicting on their employees that doesn’t really change anything. It’d be like if ExxonMobil didn’t allow their employees with company cars to fill up at a Chevron station.
It’d be like if ExxonMobil didn’t allow their employees with company cars to fill up at a Chevron station.
That is likely very much the case. When you drive a company vehicle, you have a fuel card for fill-ups that is for a particular chain and doesn’t work anywhere else.
Yeah I don’t think this is the best analogy, but the point being is brand loyalty can only go so far. Like if you’re going to run out of gas in the next 20 miles and there isn’t an Exxon station within 100 miles, do you just pass all other gas stations and have your employees break down on the side of the road?
I just can’t imagine any actual competitors to AWS would impose such restrictions on their employees that put them in a worse position to do their jobs, so it’s a bit silly that it’s coming from Walmart, when they don’t compete in that space.
I brought down all my department’s services to take a day off and blame it on Amazon. Next year when negotiating a raise/budget increase, I’ll point to this incident and take credit for migrating us off AWS after six months of in-person training classes (either in places I haven’t visited or would like to see again) and another six months of hard work in the office (napping in the server room).
2026 is looking pretty good already and I definitely won’t regret tempting fate by saying that.
Legendary.
great idea! show us your nap setup! sitting or lying down?
It’s sad that none of these people you deal with don’t know what service they use or vendors they deal with? That sounds fantastic for you, scary for everyone else.
The really stupid thing is that even if you weren’t in AWS east us 1 you were still boned because that is where AWS does it’s service authentication.
I love it when Cloud companies pretend there are “serverless” services that are “location-transparent”
You know, they sell this crap to governments and have to follow compliance regimes like FedRAMP but yet… this happens
But the only way to do this is to have a CSO willing to invest heavy in red-teaming – for attacks of every kind the team can brainstorm – and a CEO willing to spend the $$ and attention to get their recommendations implemented.
I just want AWS, Azure, Google, & Cloudflare to go down at the same time.
And maybe stay down.
If a nation wants to go to war with the US … this is how they do it, they just shut down one, two or all of these systems down and watch the country go crazy. It wouldn’t destroy the country, just disrupt it enough to make them go nuts and then do more things to them in other ways.
It’s amazing when you think about it, first the US invested in heavily defending and arming itself in the 60s, 70s and 80s … then it spent billions more in the 90s and 2000s to try to come up with ever more inventive ways to screw itself from the inside.
No. Everyone always focuses on the flashy stuff like datacenters but the truth is that the most vulnerable, overtaxed, and underfunded weak-spot for the United States is the electric grid.
Yup. Most of our core electrical infrastructure is over 100 years old now. And thanks to a combination of NIMBYism, profiteering, and the anti-nuclear brigades, we’re not likely to see that change any time soon.
I’m pretty convinced that if we did have a complete grid failure, we wouldn’t be able to complete a cold start.
Too many people would be pushing for their section to be started first so they could short the market first.
Edit: my proofreading sucks
To be pedantic for a sec, the term is black start. Just cause I think it’s super dystopian sounding.
Doesn’t matter who wants want, the power company has contingency plans and those plans are what is happening.
SOURCE: Worked for Cox after a major ice storm smashed north-central Oklahoma flat. Lived Hurricane Ivan and saw their rollout priorities.
The American dream
Note to self: Sabotage the US’ electric grid.
Which in turn would take down the datacenters too, so same effect, just more severe
Most data centers are backed up by generators.
… that run on fuel for a limited time … once the fuel runs out, the data centres go down
Need to transport more fuel there? Can’t because the entire system is down.
All*
And UPSs.
I feel like you got it backwards. Letting them run is doing more damage than turning them off would.
If the way someone goes to war with the US is by freeing us from the overly-centralized landlords of the Internet, maybe whoever they are isn’t so bad.
Cloudflare in particular can fuck off and stay fucked off.
Same, Canvas is perhaps the most used Learning Management System in the US and they apparently are entirely hosted on AWS East. The real kicker is I had my students midterm due date literally today for two classes. I’ve been swamped with panic emails (and I made clear my due dates aren’t even that important when there isn’t a national outage lol).
My head canon is someone wished for a miracle due date extension somewhere in the country and they monkey’s pawed AWS into non existence.
I thought Canvas was self hosted? Or do they offer service instances too?
I’m sure there’s a self hosted option but no way in hell it’s as popular as it is without being a turnkey solution. Most smaller districts wouldn’t touch anything self hosted with a 10 foot pole. They don’t have the resources.
I thought that too until today, it’s never clear until a quarter of the Internet is down randomly.
We all fund the devil pretending we don’t know.
Do I have any choice?
Only if you’re extremely privileged.
EBSCO was down. Ask me how my dissertation writing progressed today…
How did your dissertation writing progress today?
Half the articles I needed to read couldn’t be accessed due to aws. I figured I would grade assignments in the mean time, up until I learned e-learning was down as well. Nothing came back up till about 3.
Thanks for asking. How was your day? Was it affected by the AWS outage?
My school uses canvas for assignments, and it was down. It was pretty hectic for the teachers. Its amusing to me to see so many services down. Reminds me of the xkcd cloud. https://xkcd.com/908/
I wouldn’t look at it that way. Even companies not leveraging AWS directly will be impacted.
Especially Microsoft, Google, and Oracle, who are all at this very moment probably sending out sales droids in vast numbers
There’s so much vendor lock in with AWS, migrating to another provider over an outage even lasting 24h would be a tough sell. This isn’t unique to AWS either, each of the cloud vendors have their own lock in and their own problems. If you had the money you could run in multiple clouds, but for most businesses who were only running in a single region, I can’t imagine they’d choose this option.
Sounds like those companies are not considering the capabilities of their supply chain then.
MS-365 next!
My wife and I are at Disneyland today, and their site is down. xD
In my country, a lot of the non-traditional bank apps are still down, with millions of people having lost access to their money. Can’t even buy groceries, pay the bills, or anything.
fr i cant believe clash royale is a part of that
Yep, I just love how one service provider goes down and way to much of the internet goes down with it.
I haven’t been able to use Flickr for 10 hours. I’m so mad
scums
Not a plural/countable word.
*Amazon’s cums
One scum. Two scums. Three scums. Four scums.
Seems countable to me.
A summation of scums
Scummation or perhaps scumsock
