Recent Spammer Influx Warning. (Non-Spam Thread)

Recent Spammer Influx Warning. (Non-Spam Thread)
by Hamtaro126 on 2011-09-07 (#83607)

Recently, A new bunch of Spammers had Spammed up the site again...

This needs to be taken care of again due to possible issues of external links that may possibly lead to a scam, malware, or even a virus/worm, but in most cases, the links are just there to be taken care of immediately.

Question: In what action, may we suggest, to stop most of it lately?

Answer: Most of the NESDEV Staff in my standpoint recommend the Sign-In Protection Scheme.

I do not know if this is right from a hearing point, but letting the people not only ''Sign in'', but to verify a person by putting in two or three randomly-generated ''Nintendo-related'' words via an Image, Followed by you typing it in...

Anyone want to test this out? I'd like to help as much as possible...

Any other Suggestions?

by 3gengames on 2011-09-07 (#83608)

My suggestion is to learn not to click bot links and wait for the mods to clean it up.

by Dwedit on 2011-09-07 (#83615)

Let established members (20+ posts, and 2 weeks membership) report spam, and have it get auto-removed from the public once several people (maybe 3 to 5) report it. Then a moderator can confirm that the spam should be deleted, or reject the reports and make the post visible again. I doubt anyone here would abuse such a feature, but just in case, revoke the report feature to anyone who abuses it. Flamewars aren't common here.

by Hamtaro126 on 2011-09-08 (#83623)

I do not really click on these spam links, nor care about them.

And yesterday, the fact that I saw 3-4 spam links by a couple of spammers... not really the result of bad reports,

Next time, I will make sure you guys handle it, I am just not trying to make the forums a spam battlegrounds like some forums.

by tepples on 2011-09-08 (#83624)

Dwedit: I'm not sure community moderation like that is possible in phpBB 2. What was the big reason against upgrading to 3 again?

by tokumaru on 2011-09-08 (#83627)

tepples wrote:
What was the big reason against upgrading to 3 again?

The possibility of breaking everything?

by koitsu on 2011-09-08 (#83638)

I thought we've covered this topic many times over already? Or at least *I* thought *I* have. Oh well, here I go again...

Anonymous posting isn't permitted on the board any more. You must have a login. Signing up requires a valid Email address (for verification). But this doesn't matter -- the accounts you've been seeing post spam ARE CREATED BY HUMAN BEINGS. Software is not being used to create and validate the accounts.

There are companies in Asia and India which exist solely to do exactly that -- create accounts on forums, then proceed to sell the account credentials to the highest bidder, so the bidder can begin using them for spam. There's an alternate method of business, which involves a customer paying one of these companies to actually sign up on forums and post links to products, put a URL in their profile which people will click, etc.. The companies have employees/contractors who read or speak English.

As such, any form of captcha is effectively pointless. So please don't propose anything like that.

Also, they have an additional advantage being in another timezone; while they're awake the forum mods here are asleep. It won't be until later in the day that the spams get removed.

And worse, some of these companies use compromised machines to siphon their requests, so blocking them by IP address doesn't always work either. Welcome to the Internet in 2011.

A "voting system" for posts would be absolutely pointless as well. Ten people mark up a post as spam; what should happen to it? What if a bunch of dickheads register accounts and start marking random posts as spam (yes this will happen, and the repercussions could be devastating).

If I were to disclose how I've been dealing with the spammers, all they'd do is read this post/thread and figure out new ways to get around it. The simple version, without giving away details, is that they're being blocked gradually. It takes time. Believe me, nobody is just sitting on their ass about this issue.

Really, the spam count around here is incredibly low compared to the rest of Internet forums, my Email box, etc... I'm not saying it's not annoying, I'm saying the *scale* of annoyance is quite low compared to the rest of what's online. Settle down, bud!

by Dwedit on 2011-09-08 (#83640)

I recommended the voting system for Established Members Only, people who have been here a while and posted a lot.

by tepples on 2011-09-08 (#83641)

koitsu wrote:
Also, they have an additional advantage being in another timezone; while they're awake the forum mods here are asleep.

Do we have any regulars in Japan whom we could trust as global mods?

Quote:
A "voting system" for posts would be absolutely pointless as well. Ten people mark up a post as spam; what should happen to it? What if a bunch of dickheads register accounts and start marking random posts as spam (yes this will happen, and the repercussions could be devastating).

To expand on what Dwedit said: Look at how Slashdot has handled this. Each comment has a score from -1 to 5 (most comments start at 1), and each user has a score from -10 to 50 (start at 0). Any account that is at least a year or two old, has a high enough score, and has a certain amount of recent activity will occasionally get a package of "mod points" that can be used for increasing or decreasing the score of a comment in any thread to which the user hasn't already posted along with the score of the user who posted the comment. (Site administrators get unlimited mod points.)

Quote:
If I were to disclose how I've been dealing with the spammers, all they'd do is read this post/thread and figure out new ways to get around it.

Good. You don't need to spill the BEANS.

Quote:
Really, the spam count around here is incredibly low

I agree. Perhaps the few that get through are so noticeable precisely because the count is so low.

by Gilbert on 2011-09-08 (#83645)

Maybe a first step is to provide users who have rights to edit the Wiki the power of marking posts as spam? They already had to ask for permission for the editing rights and had their requests reviewed so basically we can consider them all trusted users, right?

by Banshaku on 2011-09-09 (#83663)

tepples wrote:
Do we have any regulars in Japan whom we could trust as global mods?

I still browse nesdev every day briefly but I'm quite busy these days. I could always see what I can do. I still check every day if any user get created on the wiki that could been a potential nesdev user and email them when it's the case. But I don't have access access to the internet as much as before thought.

by koitsu on 2011-09-09 (#83680)

tepples wrote:
koitsu wrote:
A "voting system" for posts would be absolutely pointless as well. Ten people mark up a post as spam; what should happen to it? What if a bunch of dickheads register accounts and start marking random posts as spam (yes this will happen, and the repercussions could be devastating).

To expand on what Dwedit said: Look at how Slashdot has handled this. Each comment has a score from -1 to 5 (most comments start at 1), and each user has a score from -10 to 50 (start at 0). Any account that is at least a year or two old, has a high enough score, and has a certain amount of recent activity will occasionally get a package of "mod points" that can be used for increasing or decreasing the score of a comment in any thread to which the user hasn't already posted along with the score of the user who posted the comment. (Site administrators get unlimited mod points.)

All this does is obfuscate and add unnecessary complexity to something which doesn't need it. Speaking strictly about the above "mod point" concept applied to this board:

1) People will quickly run out of mod points. Example: 20 spammers in a month. Uh oh, Hamtaro1234567890 is out of mod points within that month, and Ubernesdevdude who usually visits 5000 times a day (and has tons of mod points) happens to be on vacation in Jamaica. What now?

2) What happens to a post that gets modded down? Does it move to another forum?

3) Existing accounts cannot *safely* have mod points given to them. There are 3545 accounts on the forum:

Code:
mysql> select count(*) from phpbb_users;
+----------+
| count(*) |
+----------+
| 3545 |
+----------+
1 row in set (0.00 sec)

Guess how many of those have logged in within the past YEAR?

Code:
mysql> select count(*) from phpbb_users where user_lastvisit > unix_timestamp('2010-09-09 16:22');
+----------+
| count(*) |
+----------+
| 733 |
+----------+
1 row in set (0.02 sec)

3545 - 733 = 2812 accounts which haven't been logged into in the past year, but are probably still valid and could be used for whatever nefarious purposes.

Wait a minute, why would someone just log into them suddenly if they haven't been used in the past year? Who would do such a thing? The answer to that is implied in the next item:

4) I've been noticing spammers creating accounts on a daily basis -- usually multiples per day -- yet never posting anything. Our way of blocking spammers DOES block them from creating accounts AS WELL as posting, so the fact they're able to create accounts at all means they aren't being blocked and are therefore going to be able to post a post. They just haven't yet.

So, these spammers have a ton of usernames/passwords which they can sell to the highest bidder, or log in any time they want and use mod points (assuming we end up assigning mod points to existing accounts) to mod-up their own spams.

There's really no solution to this problem. I'm still waiting for Hamtaro389429835934 to provide his insights on how to solve this problem, since he's apparently fed up with the situation.

And as for upgrading to phpBB3 -- if that's going to happen, we need to "start over". The migration between major versions of phpBB never, EVER goes smoothly. We don't want to lose any of the existing posts (users/visitors have already stated they want those kept).

by koitsu on 2011-09-09 (#83682)

By the way, some other statistics worth pointing out:

Number of accounts which have never posted a single thing:

Code:
mysql> select count(*) from phpbb_users where user_posts = 0;
+----------+
| count(*) |
+----------+
| 2072 |
+----------+
1 row in set (0.02 sec)

Number of accounts which have posted only once and only twice, respectively:

Code:
mysql> select count(*) from phpbb_users where user_posts = 1;
+----------+
| count(*) |
+----------+
| 289 |
+----------+
1 row in set (0.02 sec)

mysql> select count(*) from phpbb_users where user_posts = 2;
+----------+
| count(*) |
+----------+
| 202 |
+----------+
1 row in set (0.01 sec)

So uh, yeah... These numbers kinda speak for themselves.

by tepples on 2011-09-09 (#83683)

You might want to take a look at the "Comments and Moderation" section of the Slashdot FAQ as well as the history of how Slashdot got to the moderation system it uses.

koitsu wrote:
1) People will quickly run out of mod points. Example: 20 spammers in a month. Uh oh, Hamtaro1234567890 is out of mod points within that month, and Ubernesdevdude who usually visits 5000 times a day (and has tons of mod points) happens to be on vacation in Jamaica. What now?

Slashdot mod points expire after a few days.

Quote:
2) What happens to a post that gets modded down? Does it move to another forum?

The user can set two thresholds. Posts with a score >= the higher threshold get displayed in full. Posts with a score >= the lower threshold display only the author, subject, and first line (after any [quote] elements have been stripped out). Posts with a score below the lower threshold display as "1 hidden comment". Users with JavaScript turned on can click a hidden or abbreviated comment to expand it to a full view.

Quote:
3545 - 733 = 2812 accounts which haven't been logged into in the past year, but are probably still valid and could be used for whatever nefarious purposes.

But do they also have enough of a post count (let's say 50) to qualify for mod points? The 2,500-odd accounts with two or fewer posts certainly don't.

Quote:
Wait a minute, why would someone just log into them suddenly if they haven't been used in the past year?

Slashdot gives mod points only to users who have been active recently.

Quote:
So, these spammers have a ton of usernames/passwords which they can sell to the highest bidder

Only if a spammer has guessed the username/password, or the user has sold the account to a spammer.

Quote:
And as for upgrading to phpBB3 -- if that's going to happen, we need to "start over".

Then we might have to handle it like the transition from wwwThreads to phpBB 2: start a new board, disable creation of new topics on the old board, and eventually disable replies on the old board.

by koitsu on 2011-09-09 (#83689)

I finished reading the story of how Slashdot's moderation system came to be and its history. I didn't skim it, I read it. I did skim the FAQ however.

There is absolutely no mention of how they deal with actual spam (the kind we get here). The term "spam" throughout the document and the FAQ means "comments from a user that really don't amount to much value" -- the #1 cited example of spam is a "first post!!!" comment. This is not the kind of spam we get here.

Their karma system is intended to let moderators "rank" users based on whether or not what they say is worthwhile. Furthermore, I see absolutely zero indication that actual spam is *deleted* on Slashdot. Surely that shit gets deleted -- in fact, I KNOW it gets deleted. Someone is deleting it. It doesn't just get "modded down" and the user given bad karma -- I have seen spam show up on Slashdot and then the next day is completely gone (even with visibility sliders set to show everything possible). So someone there is having to do the same thing we do: delete the utter crap that comes in.

Finally, regarding the visibility slider situation -- which for those who aren't familiar with it, does exactly what tepples describes: it lets you define what sort of posts (based on their modded value) you see. This also doesn't solve the problem, it just masks it. It also introduces a horrible situation that really doesn't bode well (my opinion): Hamtaro83828342 starts using the new system, realises all the spam gets modded down, and adjusts his slider so he doesn't see modded down posts. Then suddenly he reads a thread and finds that certain posts are "missing" from the thread -- relevant posts to the subject matter. He then has to adjust the slider so he can view said posts. But now he's going to see the boatloads of spam he complained about in the first place. So he's going to spend more of his time fiddling with a slider in a web GUI than he is reading actual content. :-)

Next there's the issue of a new user who signs up and has highly useful things to say. Someone's slider settings may be filtering him out entirely. He won't get karma or mod-up points on his comments because nobody's seeing what he has to say, because of the spam. Sad panda situation.

Then finally there's the issue of efficiency: more accounts = only slightly slower board/forum. However, more posts (e.g. spams which aren't getting outright deleted) = *much* slower board/forum. "Wow, nesdev has over 50,000,000 posts! ... Oh, wait, 49,500,000 of them are spam..."

All this said, I am absolutely open to suggestions, and I appreciate the time people spend to propose suggestions. Honest. The only reason I'm highly critical of them is because -- and this is stated quite boldly in the Slashdot moderation story too! -- there's a balance that has to be achieved.

Furthermore, I think tepples and I would have to be the two who ended up coding the enhancements into the board (phpBB 2.x). This makes software upgrades a bitch cuz there's custom patches. Heck, we already have custom patches to make the board work properly with UTF-8, and our version of phpBB 2 is quite old/outdated (the Admin panel reminds me of this every time I use it. :-) ). We *should* upgrade, but oh god...

Anyway, sorry for the long-winded rant, these are just the thoughts that come into my head. I have no problem with additional forum moderators being made and so on, but then there's the issue of what happens if someone starts deleting posts / abusing their status -- we do MySQL backups daily, so reverting/restoring the effects of abusive behaviour is possible, but it's a tedious process.

I really hate the idea of having to manage people. I really do. I like everyone to just be reasonable; the Parodius FAQ is sort of an example of how I feel about things. "Just don't be a dick" sums it up (and that's coming from a guy who usually *IS* a dick), but not everyone sees eye to eye.

As far as trying other forum software, we can absolutely do that, and I can spend the time to set that up *in parallel* to this board still operating. We can try softwares, let people post stuff, mess about, etc. and see what sorts of havoc can occur. Think of it as a sandbox test to see what forum software might work well for us.

Oh, and I'll state this point blank: if there's **commercial** forum software people want or like (that runs on UNIX, under PHP or Perl -- please not ASP or Mono/.NET or anything nonsensical like that), let me know, I will be happy to purchase a license for the board.

by Banshaku on 2011-09-09 (#83690)

I will add one comment regarding that slashdot related "way of doing things", which may not have much value but I will tell it anyway.

Since I always browsed slashdot lightly (recently) and I don't remember when the modding system started, I always felt that the system was just always getting in my way. What I mean is, now that I understand how it work, WHY do I have to let others decide what is supposed to be worth reading or not? I was always stuck to expand everything manually (didn't know about that option just mentioned before) and it was a pain in the butt. I just wanted to read every possible comments no matter what.

In brief, I don't want other people to decide what is supposed to be relevant or not to a thread. In a small community like ours, every mail could have some value, maybe except in the general forum. We have a small spam issue and that modding thing feels, I don't know how to say properly, a ego thing that I mod you down because I don't like what you said. In a small community like ours, it would just look like bitching at each other. It doesn't fix spam either.

Now back on the subject, there is no perfect way to handle spam. It will require someone that is willing to judge if the reason to write on nesdev is valid. I know people don't like to judge but the internet have changed unfortunately.

This mean we could require either a PM from the new user or at first they see the "welcome" forum where they must explain why they created their account then after they will be promoted to a normal user. This forum cannot be seen by bots (private) and normal users so any spam will not be indexed in search engines. There could be an invite system from well known user so people could be promoted faster too.

I know it's hard to judge someone from the get-go and you can let it some true spammer if they know what to write for their introduction but it would help filter the quick human spammer which can only post in the welcome forum. For the wiki this is easier since usually, if you want to write on the wiki, you are usually a nesdev user. I may be missing some potential user but since it's quite a niche subject, the risk is small. If that person really want to write on the wiki, they will figure out how to get in touch with nesdev first. And usually just by the user name, it's often quite easy to guess if it's a spammer on the wiki.

Some people may say "but that will delay new users to post! Blasphemy!". With the amount of post that we receive everyday... I think new users can understand that we just do that so the content on nesdev is at the least free of spam. It's not like we have 10000 new users with 1 millions posts everyday. In that case, it's a different ball game.

By the way, the count about mail, does it include PM too? Some people only do PM (or so I heard).

by koitsu on 2011-09-09 (#83698)

Your opinions/sentiments about Slashdot pretty much mirror my own, Banshaku.

Back to phpBB stuff:

The counts shown were for actual forum posts (what you call "mail"), not private messages via the forum. There is no counter in the SQL table schema for number of PMs sent or received, only number of new PMs, unread PMs, and the last time a PM was sent (or read; not sure which).

If there are users who only use the PM functionality here, they would have to be classified as "users who have logged in within the past year". Otherwise there's no way to determine if they're an "active user" other than looking at the last time they logged in to the site.