google be like stop bypassing paywalls and ads using our cache

Here’s the heart of the not-so-obvious problem:

Websites treat the Google crawler like a 1st class citizen. Paywalls give Google unpaid junk-free access. Then Google search results direct people to a website that treats humans differently (worse). So Google users are led to sites they cannot access. The heart of the problem is access inequality. Google effectively serves to refer people to sites that are not publicly accessible.

I do not want to see search results I cannot access. Google cache was the equalizer that neutralizes that problem. Now that problem is back in our face.

From the article:

“was meant for helping people access pages when way back, you often couldn’t depend on a page loading. These days, things have greatly improved. So, it was decided to retire it.” (emphasis added)

Bullshit! The web gets increasingly enshitified and content is less accessible every day.

For now, you can still build your own cache links even without the button, just by going to “https://webcache.googleusercontent.com/search?q=cache:” plus a website URL, or by typing “cache:” plus a URL into Google Search.

You can also use 12ft.io.

Cached links were great if the website was down or quickly changed, but they also gave some insight over the years about how the “Google Bot” web crawler views the web. … A lot of Google Bot details are shrouded in secrecy to hide from SEO spammers, but you could learn a lot by investigating what cached pages look like.

Okay, so there’s a more plausible theory about the real reason for this move. Google may be trying to increase the secrecy of how its crawler functions.

The pages aren’t necessarily rendered like how you would expect.

More importantly, they don’t render the way authors expect. And that’s a fucking good thing! It’s how caching helps give us some escape from enshification. From the 12ft.io faq:

“Prepend 12ft.io/ to the URL webpage, and we’ll try our best to remove the popups, ads, and other visual distractions.”

It also circumvents #paywalls. No doubt there must be legal pressure on Google from angry website owners who want to force their content to come with garbage.

The death of cached sites will mean the Internet Archive has a larger burden of archiving and tracking changes on the world’s webpages.

The possibly good news is that Google’s role shrinks a bit. Any Google shrinkage is a good outcome overall. But there is a concerning relationship between archive.org and Cloudflare. I depend heavily on archive.org largely because Cloudflare has broken ~25% of the web. The day #InternetArchive becomes Cloudflared itself, we’re fucked.

We need several non-profits to archive the web in parallel redundancy with archive.org.

I think we can safely say that “web page don’t go down anymore” is not the real reason. Two possibilities I see for the real reason:

• They don’t feel it’s necessary anymore to cache the actual pages. It’s not productive to have them to track and analyze SEO abuse, or even just to tune their algorithm by rescanning things if someone notices there’s a problem, even given that it costs them a fraction of a fraction of a fraction of the income they make from search, and it makes life easier for anyone who’s working on it.
• They used to make it available just because it was a neat thing to be able to offer and there was no reason not to, since they were caching the pages anyway. But, the current Google management has wandered so far afield of the original mindset that made them successful in the first place that that means nothing to them, more like a bizarre confusion of concepts than any kind of statement that makes coherent sense. So fuck the users. Yes we’re still caching them, because the search engine needs them. No you can’t have them for free, you fucking hippie. Now get out.

I know which explanation I favor.

Google “Search Liaison” Danny Sullivan confirmed the feature removal in an X post, saying the feature “was meant for helping people access pages when way back, you often couldn’t depend on a page loading. These days, things have greatly improved. So, it was decided to retire it.”

okay but… has it? this seems like an unfounded premise, intuitively speaking

Not dead, just non-public.

AIs are hungry, the value of stored shitposts skyrocketed.

Google never backed up the internet, sure they did cache pages, but that isn’t even close to backing up the internet

This is probably an attempt to save money on storage costs. Expect cloud storage pricing from Google to continue to rise as they reallocate spending towards ML hardware accelerators.

Never been happier to have a proper NAS setup with offsite backup 🙃

Maybe they don’t want to give rival AI devs data access? It’s not typical for Google to give up data.

Is there any sort of way to self host a limited version of this? L

I’d love to be able to have my own Searx also cache everything I visit as I go to it, it’d at least let me refind information I’ve previously found.