time-out, forbidden, blocked for days at a time :(

[Galactic]

Similar to others' reports, I'm getting time-outs, forbiddens and blocked messages when accessing the forum.

For various reasons beyond my control, I have to access the web through a VPN (wide range of changing IP addresses) or the TOR network. As such, checking and whitelisting various IP address ranges won't help in my user case.

Over the last few weeks about 90% of access attempts to the forums have been prevented, particularly on my desktop (but sometimes are accepted through the mobile device, even though on same network). Consequently I've often been kept away for several days at a time, or (since I hate typing long texts on my phone), haven't contributed to the boards when I otherwise would have done.

I'm very glad the admins here haven't succumbed to Cloudflare, and being responsible for several websites myself am very pleased by your roll-your-own response to the shocking onslaught on the web by LLMs and other bots.

When it comes to blocking IP addresses I personally prefer a variable timeout rather than a permanent block on the sites I administer. I find that works well in terms of reducing traffic and server-load while reducing collateral. I wonder whether this or some other solution may be a possibility for this forum since the problem of bots sucking up bandwidth and knowledge isn't going to get better any time soon, and more and more genuine readers and posters such as myself are finding themselves on the wrong side of blocks.

I also have on my to do list the task to have a look at some FOSS bot tests which work similarly to, but better than Cloudflare and are locally hosted. As a visitor I never seem to have problems with those, no matter which IP address I'm coming in from. I wonder how well these would work with boards like this?

However you deal with this problem, I wish to thank the admins for all their work in keeping this site going over the years.

[admin]

Thank you for your comments

We might need to re-think the general blocks on VPNs. While they are used a great deal by spammers to appear to be in a different country, VPNs are being used by "normal people" more and more too.

I'm also not keen on Cloudflare and the like. They might solve some problems, but also tend to introduce new problems!

I hope that the bot onslaught will subside: as the "AI" bubble bursts, the need to slurp content from everywhere on the internet will disappear. Slow methodical indexing by search engines has never been a problem, and that should be the only real need for robot visitors.

[admin]

Just found this article, which is an interesting description of the problem:

https://lwn.net/Articles/1080822/

Interesting about the mobile-app-based "NetNut" residential proxy network. A lot of scraping IP addresses have appeared to be coming from mobile network operators...

[slowster]

Over the last few weeks about 90% of access attempts to the forums have been prevented

Prevented by what exactly?

- Time-outs?
- The message "Forbidden You don't have permission to access this resource"?
- The message "A ban has been issued on your IP address. If you are using a VPN (virtual private network), it is likely that you will be able to access the forum by turning the VPN off"?

[mjr]

I'm very glad the admins here haven't succumbed to Cloudflare, and being responsible for several websites myself am very pleased by your roll-your-own response to the shocking onslaught on the web by LLMs and other bots.

Me too. I am really tired of "cloudf'd" websites locking me out because I don't run an easily-exploitable browser, I don't accept cookies without seeing a privacy policy first and I find it difficult to pass an eye test (which is why I wear glasses to ride... who knew that reading from a computer screen arm's length away would one day require better eyesight than hurtling downhill at 30+ mph?

VPN use is only likely to increase with all the control-freaky governments about. Are there lists of which VPNs discourage or even ban use by LLM (Artificial Incompetence) scrapers? It might be worth recommending them in messages to blocked ones.

[slowster]

Are there lists of which VPNs discourage or even ban use by LLM (Artificial Incompetence) scrapers? It might be worth recommending them in messages to blocked ones.

I recommend reading the article to which admin posted a link above, which explains what 'residential proxy servers' are and how they are used by malign actors.

The simultaneous attacks by large numbers of bots seeking to scrape data from this forum are typically carried out using residential proxy servers. For example, this morning I checked the IP addresses of just a few of the large number of bots that were attacking the forum, and they belonged to users in Bangladesh, Morocco, Viet Nam and Italy.

My uneducated and ill-informed guess is that that network of residential proxy servers is being used by a country/tech giant which is trying to catch up with Google, Amazon, and other leading players in the field of LLMs. So probably Russia, China or India, or rather a Russian, Chinese or Indian tech giant, e.g. Yandex in Russia is state owned/controlled.

Whereas spammers almost always use VPNs when attempting to post on the Forum, the IP addresses of residential proxy servers are ordinary user IP addresses, not VPN IP addresses.

However, VPNs do play a role in residential proxy server networks, because installing a dodgy free VPN may result in malware being installed on a user's device that enables it to be hijacked and used as a residential proxy server. The article below on the IPIDEA bot network lists a dozen proxy server and VPN providers that were found to be controlled by those behind IPIDEA. It appears that anyone who has used them has potentially compromised the security of their device.

https://cloud.google.com/blog/topics/th ... xy-network

[mjr]

Are there lists of which VPNs discourage or even ban use by LLM (Artificial Incompetence) scrapers? It might be worth recommending them in messages to blocked ones.

I recommend reading the article to which admin posted a link above, which explains what 'residential proxy servers' are and how they are used by malign actors.

Yes, I work in the sector and read Brian Krebs's article linked there when he posted it. His post this week about using malware VPNs based on streaming sticks for ad fraud is pretty interesting too, especially relevant here with how many cycle racing fans have decided to try unlawful streaming instead of pay TNT's high charges. https://krebsonsecurity.com/2026/07/rea ... ing-stick/

Whereas spammers almost always use VPNs when attempting to post on the Forum, the IP addresses of residential proxy servers are ordinary user IP addresses, not VPN IP addresses.

That makes it sound like your advice is for people to buy access to these residential proxies! Seems a bit irresponsible so I hope that's just a poor choice of what to mention in an answer to a question about what non-IT-specialist people could use.

[Galactic]

Over the last few weeks about 90% of access attempts to the forums have been prevented

Prevented by what exactly?

- Time-outs?
- The message "Forbidden You don't have permission to access this resource"?
- The message "A ban has been issued on your IP address.

All of the above (obviously not all at the same time)

Thanks to @admin for the link to the LWN article, which provided yet more (disturbing) food for thought. We live in sad times.

(Apologies for delay in responding, this was due to the 'all of the above' )

[slowster]

All of the above (obviously not all at the same time)

I asked because although they all affect your ability to use the forum, they are different and need to be considered separately.

1. I get time outs occasionally which are resolved simply by refreshing the page. However, sometimes this does not work and it may be five or ten minutes before I can access the forum/a page on the forum. I think this can be more likely to happen when the forum is being aggressively scraped by large numbers of bots, slowing the server down.

I only realised relatively recently that the information at the bottom of the index page can indicate if the forum is being subjected to a bot attack. As I write this there are 527 'guests' browsing the forum. Many of those are bots. In other words the scraping is happening almost all the time (this is in addition to the relatively standard/benign bots used by the major search engines to index the forum). Occasionally the scraping/bot attacks get much worse and the forum performance is noticeably degraded. That now prompts me to check the number of guest users to see if they are the cause. The last time that happened there were some 2,000 guest users. admin presumably gets automatic notifications when the server is being overloaded by an attack.

You will also see at the bottom of the index page "Most users ever online was 53441 on 17 Jul 2025, 10:05am". That gives you an indication of the potential severity of the attacks. Prior to that it was 'Most users ever online was 7022 on 14 Mar 2025, 3:41pm' and the number used to change roughly every month or so with a newer and slightly higher number.

admin has explained how he is dealing with these attacks in various posts, e.g.
- viewtopic.php?p=1943812#p1943812
- viewtopic.php?p=1955688#p1955688
- viewtopic.php?p=1951263#p1951263

2. I think you are probably seeing "Forbidden You don't have permission to access this resource" when you use a VPN exit node in a country which admin has blocked from accessing forum members' user profiles. admin did this because those were being heavily scraped by AI bots. The block was initially much wider than now, and admin has added back in a few countries in response to queries from members in those countries.

If it is user profiles that you are trying to see, and you are using a VPN, I expect you will continue to see "Forbidden You don't have permission to access this resource". I don't think access to other members' profiles is something that is essential for members, e.g. it does not prevent sending someone a PM or seeing their posting history.

Edit - In addition to a narrow block on accessing user profiles, I think you might also see the message when admin has responded to a major bot attack by completely blocking access based on country or ASN, as detailed in the first of the links above, i.e. again if your VPN exit country is one that has been blocked by admin.

3. If you see the message "A ban has been issued on your IP address. If you are using a VPN...", you are almost certainly using a VPN IP address which belongs to a range which has been used by spammers who have attempted to post on the forum. Unlike 1 and 2 above, those blocks have been added by me using the forum's own software function for banning IP addresses, bad email domains and individual users.