Doesn't really work if crawlers are coming from the IP ranges of AWS and Azure etc...

Traditionally, holders of IP ranges that attack the internet at large get kicked off the internet by having those ranges blacklisted everywhere. This can also get them in serious trouble with the places they got their IP ranges (I assume AWS has them directly from ARIN, so maybe not) and their upstream bandwidth providers and so on, as well as making them less attractive hosts because they are blocked everywhere.

Kicking AWS of the internet would effectively break half of everything. So much stuff is relying on services running on AWS that it's not even funny.

We've seen random stuff break when AWS has had outages, not because we used AWS ourselves, but because suppliers do.

That's actually an argument in favour of kicking AWS off the Internet. We rely too much on their services, to the point we're afraid of banning their IPs if they do something bad. Better stop this now than being worse off later. The best moment would have been ten years ago, the second best moment is today.

You don't need to kick anyone off the internet. Just ban them for accessing your resources.

No, but the suggestion from the parent comment was to have the holder of the offending IP ranges kicked off the internet.

Technically I'm all for kicking AWS off the internet for a day or to, for failing to police their customers, but it would just break a lot.

AWS and Azure can be blocked like anyone else.

Or sometimes they use consumer IP proxies. Makes it even harder because sometimes those IPs get reused for actual users.

Block them for 24 hours.

just ban the lot

nothing good comes from there

unfortunately then they instantly switch to home IPs

There's already loads of these. The problem is that most of these IPs are just cloud providers or DC ISPs.

Or even worse, lots of them are using barely legal residential proxies so the requests are coming from everywhere. In Drew DeVault's article linked in this post he complained precisely about the residential-looking source IP addresses [0]. And I think I remember something about a Chinese company, some months ago, very aggressively scraping using that method.

Companies like DataImpulse [1] or ScraperAPI [2] will happily publicize their services with that specific target.

--

  0: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html
  1: https://dataimpulse.com/use-cases/ai-proxies/
  2: https://www.scraperapi.com/solutions/ai-data/

Are these "residential proxies" assumed to be infected devices part of a botnet or malicious apps on users' phones?

A lot of these residential proxy as a service companies just use regular ISPs to run their automated headless browsers. It doesn't need to be illegal.

I guess iot devices and mobile apps.

Unethical, definitely. Illegal, no.