Regarding the matter of noisome spam

[SparkyChick]

Unfortunately that wouldn't be possible as the forum has to cache each page so serve the same to everyone who requests it. The server doesn't actually receive a good proportion of the requests as that would make the running of the site a lot more expensive. Only when someone makes a postp the cache would be flushed and then there would be an update to the cache.
I used to be in a team running a well known website and the bill for back end servers alone was thousands a month and the same for bandwidth on top.
Even a small tweak to the caching to improve it saved us 1k a month in bandwidth, so it is important.

Question... how do you cache a page that is unique to every user each and every time they request it (unless they haven't actually read any posts and no new posts have been made)?

My next question... define well known (how many visitors, daily page serves, that kind of definition) and if you wouldn't mind... how many web servers, database servers etc.

 

[static zap]

I'm still suspicious about the "Metrics" used ,
The spam must be putting some off , but does it make site look good , click wise ... as that's not a good situ !

 

[Marvo]

Does all the posting they make count as traffic for the site?

Interesting question but looking at the figures on Alexa about where the traffic is coming from, I don't think they do. Electriciansforums.co.uk Traffic, Demographics and Competitors - Alexa - https://www.alexa.com/siteinfo/electriciansforums.co.uk

 

[Marvo]

Question... how do you cache a page that is unique to every user each and every time they request it (unless they haven't actually read any posts and no new posts have been made)?

My next question... define well known (how many visitors, daily page serves, that kind of definition) and if you wouldn't mind... how many web servers, database servers etc.

You don't cache unique pages such as search results etc or at least not for any significant period. Pages such as popular threads are cached both at server level and at CDN level by Cloudflare. There may also be load balancers etc that hold cached pages but I'm not sure with this particular site.

 

[Bobster]

Interesting question but looking at the figures on Alexa about where the traffic is coming from, I don't think they do. Electriciansforums.co.uk Traffic, Demographics and Competitors - Alexa - https://www.alexa.com/siteinfo/electriciansforums.co.uk

I guess they could be using a vpn. Especially with China's internet laws. However I'm also sure you can tell from their IP address if they are or aren't.

Was just curious.

 

[johnduffell]

Question... how do you cache a page that is unique to every user each and every time they request it (unless they haven't actually read any posts and no new posts have been made)?

You actually cache responses rather than pages in reality, so all the styles and images and whatnot is cached for a while. The response will be basically the high level and structure for the page itself, as well as the content. I.e. the html. And the "new posts" page response will be served identially for all users until a post is made, then a new version will go to all users, etc. If the server started making bespoke responses depending who's ignored who it would grind to a halt more quickly.
Looking at the response for the last hop it's marked as not cacheable, so I suspect there's some edge side includes going on but that's another topic and probably not relevant to this thread.

My next question... define well known (how many visitors, daily page serves, that kind of definition) and if you wouldn't mind... how many web servers, database servers etc.

last I checked it was around 100million browsers a month and 30 million plus page views from up to 10 million browsers a day although that includes apps as well as web, IIRC we were serving hundreds of GB per hour from the origin and that doesn't include images or videos, css/js etc. Pretty much just html and ajax responses. Database was done by another team, web servers was around 30 although some of them were busier than others.

 

[SparkyChick]

I know how web caching works... the point I was really getting at is that forums present a unique challenge when it comes to caching.

Pretty much every page is unique to every visitor at any particular point in time since their content will typically depend on a set of state data that is unique to that user at the time the page is served.

My new posts page is unique to me because the content depends entirely on what's been posted and what I've read. Other content is changing constantly as well, like post counts, on/off-line status, post ratings, trophy points etc.

So, I stand by my statement that the best way to handle the new posts page is to move the ignore processing into the database. It's then being handled by a piece of software designed for such purposes (i.e. the relational database management system). The scripts which generate pages are built in a general purpose computing environment which does lots of things well (compared to the RDBMS which does a few things incredibly well).

So using this (obviously this is meant as an example)...

SELECT * FROM POSTS WHERE (NOT (USERID IN (1,2,3,etc.)))

That is the bread and butter job of an RDBMS.

Doing that has the potential to remove (if there are 200 new posts in the new posts page) 200 conditional branches (each of which has to iterate fully over what could be a large list of user ID's to be ignored). It also solves the problem of lots of spammers being ignored = empty new posts page because ignored posts won't feature in the record limit of the query (which I believe is 200 records).

I've spent a great deal of time optimising complex systems in order to improve their performance. Making the best use of the tools at your disposal is the first step. Use specialist tools rather than general purpose compute as in the vast majority of cases you can achieve marked performance enhancements (and in this case solve a bug), and quite often with minimal work.

All I was trying to do was explain why the issues with ignoring lots of spammers results in an empty new posts page if the spammers have been busy and have filled it with their crud.

RE: the system you mentioned, thanks for the info. Always curious about things like that.

 

[123]

@SparkyChick I was about to say exactly the same thing.

(I have no idea what you are on about)

 

[johnduffell]

Pretty much every page is unique to every visitor at any particular point

True, but if the fundamental data is the same, then you can cache that. We have a discussion system and although that has the most requests, they can be cached pretty well because it's all iterated through with ajax. For every post and thread it is read many more times than it is written.

the best way to handle the new posts page is to move the ignore processing into the database.

Depends on your definition of best, I would have done the same before working on that site. But sending more requests to the DB is going to kill your scalability. And database is very expensive compared with runnning some JS to stitch things together on the client side. That's kind of free once you get away from dev costs. A clue when it's happening here is that the page loads with a bare header and after a few seconds the New posts icon goes red and the flag for new posts.

I don't doubt your experience, and I agree that different sites have different needs, in my case it's an in house written system, so I could design and implement things myself. I made some changes in this vein, two were saving >1k per month by adjusting the way we did caching and one was a rewrite related to stopping having to scale the servers up loads when frequently changing forum style content was active to avoid outages. Specifically related to reducing the number of ways we were serving the same content in different ways.
So I can only speak from those experiences. The system I work on nowadays I recommended we not to worry about caching due to the different needs. Developer time is the main expense there and caching adds complexity.

All I was trying to do was explain why the issues with ignoring lots of spammers results in an empty new posts page if the spammers have been busy and have filled it with their crud.

Yep that's a good point and I didn't mean to disrupt that so sorry, I just wanted to help move forward the discussion in case it helps people's understanding (the mods obviously don't want people to feel like the problem is being ignored, actually it's tricky especially for off the shelf software)

 

[static zap]

Just re write Shakespeare
... To be OR NOT ...
is computer programmer territory.
( I'm suitably interested - Does not feel like an "English lit lesson on a hot day ")
More reads than writes ..Like

 

[Dan]

Hi all,

Not been ignoring the issue, just have been testing some things on the forums before implementing it across all of the network.

It seems the spammers are registering using humans. Then using bots to physically post to the forums.

So I've now changed the way new members are managed and checked, so the mods might need to approve some posts from time to time before they go live (newer members only) but it's certainly going to be less work than erasing the amount of spam we were getting.

I'll give it some time for the spammers to sod off and then will relax the settings so less genuine posts sit in the mod queue.

Thanks for your patience.

 

[Pete999]

Hi all,

Not been ignoring the issue, just have been testing some things on the forums before implementing it across all of the network.

It seems the spammers are registering using humans. Then using bots to physically post to the forums.

So I've now changed the way new members are managed and checked, so the mods might need to approve some posts from time to time before they go live (newer members only) but it's certainly going to be less work than erasing the amount of spam we were getting.

I'll give it some time for the spammers to sod off and then will relax the settings so less genuine posts sit in the mod queue.

Thanks for your patience.

Personally I have had some success by ignoring each and every spammer, although I don't see the post I still get number of new posts, but that's easy to overcome, just delete all new posts, but check by going through the pages, just in case you delete a post you are interested in, not ideal, but it does rid your screen of all that Chinese or whatever crap it is.

 

[Dan]

Bless you for going through all that!

We've got it covered now, at least for the time being. The swines will eventually somehow get around it no doubt but that's for another day.

 

[Pete999]

Bless you for going through all that!

We've got it covered now, at least for the time being. The swines will eventually somehow get around it no doubt but that's for another day.

Fingers crossed Dan, hope it works, it's real boomer isn't it.

 

[ruston]

Anyway, you never know when you might need a masters degree from some obscure American university written in Chinese sometime.

Or an 18th edition ...... . Payback time

 

[telectrix]

shucks. and i really wanted a fake BSc. from the university of Hai Wong Fung.