Author Topic: Chinese search engine spidering my site.  (Read 3156 times)

0 Members and 1 Guest are viewing this topic.

Offline Scooter Trash

  • Stadium Superstar
  • ******
  • Posts: 3035
  • Good Vibes 80
Re: Chinese search engine spidering my site.
« Reply #15 on: September 11, 2012, 12:12:37 pm »
Speaking of SEO stuff, you know about http://www.selfseo.com/ ?
Has some handy tools.

Wow! That looks like a "one stop shopping" place for SEO. I bookmarked it. Thanks!
I have an old application called "Traffic Seeker" that I bought years ago and just upgraded.. It's not great, but it helps. I also use an SEO plugin for WordPress, and the Webmaster tools that I mentioned earlier. I still need to work on linkage, keywords, and content, but I don't have a sales-based Website so I don't worry a whole lot about it. My ranking isn't very high on Google, but most of my commonly used  keywords don't have a lot of competition so I come up at or near the top of the first page in most of the search engines... Anyhow, thanks for the link! When I get motivated to do some more SEO, I'll definitely use it!
I dream of a better tomorrow where chickens can cross roads without their motives being questioned.

Offline Dan Graves

  • All Time Legend
  • *******
  • Posts: 6558
  • Good Vibes 168
  • Is on the Outside, looking in
Re: Chinese search engine spidering my site.
« Reply #16 on: September 11, 2012, 12:32:54 pm »
This is the sort of thing where my time spent as a scriptkiddie comes in handy  ;D
Well, this and cleaning up after malware infections  8)
"You need a little bit of insanity to do great things"
--Henry Rollins

Offline Scooter Trash

  • Stadium Superstar
  • ******
  • Posts: 3035
  • Good Vibes 80
Re: Chinese search engine spidering my site.
« Reply #17 on: September 11, 2012, 01:27:12 pm »
This is the sort of thing where my time spent as a scriptkiddie comes in handy  ;D
Well, this and cleaning up after malware infections  8)

I'm hoping that the people I'm blocking don't take it too personal... Most of them are bots, so I don't think they will..
I dream of a better tomorrow where chickens can cross roads without their motives being questioned.

Offline Dan Graves

  • All Time Legend
  • *******
  • Posts: 6558
  • Good Vibes 168
  • Is on the Outside, looking in
Re: Chinese search engine spidering my site.
« Reply #18 on: February 17, 2015, 10:28:59 am »
I seem to be digging up threads like it's going out of fashion, but what the heck...
Ever since i started running a blog on one of my Raspberry Pi's i've been running several anti-spam tools, and at some point last week the amount of crap that was flooding the poor little Pi was so bad i had to take it offline.
Now, since this blog is run off my second home connection, i obviously don't have a hosting provider who can blackhole all the bad traffic...
So i decided upon an experiment : locally blackholing China (including Hong Kong) and Russia via mod_geoip...
Spam incidence down to 1% of it's original volume.
Then looked up the IP's for the remaining spam traffic : all from Turkye and/or TOR.
Blackholed those as well : no more spam.

It's rather sad that one has to go to such measures to keep a clean blog, but it beats nuking the spammers from orbit, eh ?
"You need a little bit of insanity to do great things"
--Henry Rollins

Offline m_c

  • Arena Rocker
  • *****
  • Posts: 746
  • Good Vibes 33
Re: Chinese search engine spidering my site.
« Reply #19 on: February 18, 2015, 12:43:44 am »
I've had spammer issues in the past, and blacklisting the IP addresses provided at the link below cured 99% of the problem. The list is regularly updated with known problem IPs, and minimises the resources needed to block questionable IPs.

http://www.wizcrafts.net/chinese-blocklist.html

Offline Dan Graves

  • All Time Legend
  • *******
  • Posts: 6558
  • Good Vibes 168
  • Is on the Outside, looking in
Re: Chinese search engine spidering my site.
« Reply #20 on: February 18, 2015, 12:04:36 pm »
I'm running Nginx, and so using mod_geoip, which really doesn't slow things down much at all (loadtime difference of about half a second), but thanks for the suggestion, i'll have a peek at that list.
"You need a little bit of insanity to do great things"
--Henry Rollins

Offline Scooter Trash

  • Stadium Superstar
  • ******
  • Posts: 3035
  • Good Vibes 80
Re: Chinese search engine spidering my site.
« Reply #21 on: February 18, 2015, 12:19:23 pm »
Ideas for best option on shared server? (Go Daddy)
I dream of a better tomorrow where chickens can cross roads without their motives being questioned.

Offline Dan Graves

  • All Time Legend
  • *******
  • Posts: 6558
  • Good Vibes 168
  • Is on the Outside, looking in
Re: Chinese search engine spidering my site.
« Reply #22 on: February 18, 2015, 11:34:29 pm »
I'd still say Nginx with mod_geoip scooter, although from what i hear the best idea is to first drop Godaddy like a ton of bricks...
YMMV.

@ m_c : I had a peek at the link, but if there's one thing i've learned from working with the .htaccess file, it's that if i add a load of ip adresses in there, it'll do worse things to site performance than mod_geoip, especially under extensive load.
Mind you, that's on a low-power system like my Raspberry Pi, i've never quite messed with it on proper servers, but that's because i'm used to using Nginx and PHP-FPM on systems where i have no need for further resource saving (dedicated servers).
"You need a little bit of insanity to do great things"
--Henry Rollins

Offline m_c

  • Arena Rocker
  • *****
  • Posts: 746
  • Good Vibes 33
Re: Chinese search engine spidering my site.
« Reply #23 on: February 19, 2015, 12:09:24 am »
When I implemented the .htaccess option, it was the simplest workable option.
At the time, the server didn't natively support anything like mod_geoip, and implementing it would involve plugins and extra server resources. A list of blocked IPs in the .htaccess was an easy fix only needing occasional manual updates.

It's not something I've had to deal with lately, as I've not really been doing much webserver work, although I really need to get my own business website done at some point, when I finally decide what I'm going to use for an online shop.

Offline Scooter Trash

  • Stadium Superstar
  • ******
  • Posts: 3035
  • Good Vibes 80
Re: Chinese search engine spidering my site.
« Reply #24 on: February 19, 2015, 05:38:28 am »
I'd still say Nginx with mod_geoip scooter, although from what i hear the best idea is to first drop Godaddy like a ton of bricks...
YMMV.

Thanks Dan.

Like m_c, I've just been editing the .htaccess file and have been able to cut down on the data scrapers and pill spammers quite a bit. If I wasn't on a shared server, I'd block Russia, China, and Ukraine. I've had a decent experience with GoDaddy for about 4 years. I'll look into Nginx and the mod_geoip. Thanks again :)
I dream of a better tomorrow where chickens can cross roads without their motives being questioned.

Offline Dan Graves

  • All Time Legend
  • *******
  • Posts: 6558
  • Good Vibes 168
  • Is on the Outside, looking in
Re: Chinese search engine spidering my site.
« Reply #25 on: February 19, 2015, 11:17:38 am »
Not sure what you're running exactly scooter, but if it's Wordpress, you can locally stop the spammers from visiting your WP frontend (and backend) with the IQBlock plugin.
It relies on the same blocklists that mod_geoip uses, but unlike the more serverwide Nginx implementation, this runs just for the WP installation itself.
"You need a little bit of insanity to do great things"
--Henry Rollins

Offline Scooter Trash

  • Stadium Superstar
  • ******
  • Posts: 3035
  • Good Vibes 80
Re: Chinese search engine spidering my site.
« Reply #26 on: February 19, 2015, 06:00:16 pm »
Not sure what you're running exactly scooter, but if it's Wordpress, you can locally stop the spammers from visiting your WP frontend (and backend) with the IQBlock plugin.
It relies on the same blocklists that mod_geoip uses, but unlike the more serverwide Nginx implementation, this runs just for the WP installation itself.

I have WP and SMF and a few HTML pages.
I have plugins installed that seem to do a decent job of blocking comment spammers, etc. It's the data scrapers that I'm concerned about.. I don't really have sensitive info in the forums other than a Birthday thread for users that would allow people to get birth dates.. I'm not sure how secure the SMF database is on GoDaddy. I use SiteLock Secure, which did find some script in my theme which was developed by someone in Japan and was available on the WP theme site/page. I removed the malicious script, and the theme works fine, but now Google wants me to make it more mobile device friendly.. It's always something lol..
I dream of a better tomorrow where chickens can cross roads without their motives being questioned.

Offline Dan Graves

  • All Time Legend
  • *******
  • Posts: 6558
  • Good Vibes 168
  • Is on the Outside, looking in
Re: Chinese search engine spidering my site.
« Reply #27 on: February 19, 2015, 10:04:38 pm »
Well if you want to keep scrapers out...
The carpetbombing approach is the only way to be sure.
It sucks, but the only way to keep them away with any sort of reliable results is to blacklist the countries where the worst offenders originate, and blocking all known proxies/vpn's that are friendly to that sort of thing.
"You need a little bit of insanity to do great things"
--Henry Rollins

Offline Scooter Trash

  • Stadium Superstar
  • ******
  • Posts: 3035
  • Good Vibes 80
Re: Chinese search engine spidering my site.
« Reply #28 on: February 19, 2015, 10:06:30 pm »
Well if you want to keep scrapers out...
The carpetbombing approach is the only way to be sure.
It sucks, but the only way to keep them away with any sort of reliable results is to blacklist the countries where the worst offenders originate, and blocking all known proxies/vpn's that are friendly to that sort of thing.

Kewl.. Thanks again Dan.  8)
I dream of a better tomorrow where chickens can cross roads without their motives being questioned.

 

Get The Forum As A Mobile App