I've been out of the BBS game for close to 10 years. I use to run
Warzone BBS, my old handle was Mrproper. I'v now got back into BBSING,
Good to see you and warzone back, my situation is much the same.. :)
Thanks for the welcome, I'm not going by the name Warzone anymore, I do plan on having some doorgames, but now I'm wanting to host mainly
Amateur Radio and RC model related stuff, also I'm trying to cater to
the local community more with a pots dial in line, so I have opted for a more fitting name which I hope is in the origin line. Havent really
tested to see if anything works yet, I can get into it myself, I do
think my firewall opened correctly, but my router is weird, I do not NAT is working because all the hack attempts I'm getting via telnet (TONS compared to the way it used to be), but if I try to login using the nwga_net.synchro.net name from the local network It goes straight to my router setup, both on port 23 and 80. Does everyone else get tons of
hack attempts or is this an isolated incident? my computer is locked tighter than fort knox, but it bugs me. looks like most of them are
either trying to root the pc, or login as admin, and also give "busybox" commands from what I can tell is mainly used on android cell phones.
I get tons of it here too. Mystic has auto banned most of it so it's not such a bother anymore. Most of it seems to come from China and that part of the world, don't know what they are trying to accomplish but into the bit bucket they go. I should probably just ban china as a whole since there is no chinese content here but I've left it open so anyone there who reads english could log in and participate if they wanted too. Don't think that's ever happened but it could. :)
As much as I hate to, I've put china, russia and others into the
host filter, it seems that's where most of them originate. The
ones that are trying to gain root access to android phones seem to
all come from the US, they are more than likely drone computers or
phones that have been compromised to do the dirty work.
tested to see if anything works yet, I can get into it myself, I do think my firewall opened correctly, but my router is weird, I do not NAT is working because all the hack attempts I'm getting via telnet (TONS compared to the way it used to be), but if I try to login using the nwga_net.synchro.net name from the local network It goes straight to my router setup, both on port 23 and 80.
Does everyone else get
tons of hack attempts or is this an isolated incident? my computer is locked tighter than fort knox, but it bugs me. looks like most of them are either trying to root the pc, or login as admin, and also give "busybox" commands from what I can tell is mainly used on android cell phones.
was it on your ftp side, if so I get it also
Re: Re: Welcome back
By: Lord Time to Al on Wed May 13 2015 10:23 pm
was it on your ftp side, if so I get it also
Nah, most of it is telnet and ssh. the only thing fooling with my ftp is googlebot.
Re: Re: Welcome back
By: Lord Time to Al on Wed May 13 2015 10:23 pm
was it on your ftp side, if so I get it also
Nah, most of it is telnet and ssh. the only thing fooling with my ftp is googlebot.
Re: Re: Welcome back
By: Lord Time to Al on Wed May 13 2015 10:23 pm
was it on your ftp side, if so I get it also
Nah, most of it is telnet and ssh. the only thing fooling with my ftp is googlebot.
I had to specifically block Googlebot, it was a constant onslaught of connections every other minute for days on end. Does anyone know why Googleb just keeps crawling the FTP site endlessly?
~KenDB3
tested to see if anything works yet, I can get into it myself, I do think my firewall opened correctly, but my router is weird, I do
not NAT is working because all the hack attempts I'm getting via telnet (TONS compared to the way it used to be), but if I try to login using the nwga_net.synchro.net name from the local network
It goes straight to my router setup, both on port 23 and 80.
thats why my bbs telnet port is on 24 and my bbs http is on port 81
I had to specifically block Googlebot, it was a constant onslaught of connections every other minute for days on end. Does anyone know why Googlebot just keeps crawling the FTP site endlessly?
Re: Re: Welcome back
By: KenDB3 to kk4qbn on Thu May 14 2015 07:34 pm
Re: Re: Welcome back
By: Lord Time to Al on Wed May 13 2015 10:23 pm
was it on your ftp side, if so I get it also
Nah, most of it is telnet and ssh. the only thing fooling with my ftp is googlebot.
I had to specifically block Googlebot, it was a constant onslaught of connections every other minute for days on end. Does anyone know why Googleb just keeps crawling the FTP site endlessly?
~KenDB3
Shhhh..... government sponsored searches for mostly terror and kiddie related files.
Mickey
Re: Re: Welcome back
By: KenDB3 to kk4qbn on Thu May 14 2015 07:34 pm
I had to specifically block Googlebot, it was a constant onslaught of connections every other minute for days on end. Does anyone know why Googlebot just keeps crawling the FTP site endlessly?
googlebot is an asshole!
tested to see if anything works yet, I can get into it myself, I do think my firewall opened correctly, but my router is weird, I do not NAT is working because all the hack attempts I'm getting via telnet (TONS compared to the way it used to be), but if I try to login using the nwga_net.synchro.net name from the local network It goes straight to my router setup, both on port 23 and 80.
thats why my bbs telnet port is on 24 and my bbs http is on port 81
you guys could always get a router that isnt a POS
I can't, it'd a dsl modem with a router (from the isp - tds)
So, you're saying I should take down my anarchy related junk? LOL
I can certainly agree to that statement! It was just frikken relentless. I figured it would eventually give up but days later it was still going.
I can't, it'd a dsl modem with a router (from the isp - tds)
you guys could always get a router that isnt a POS
I had to specifically block Googlebot, it was a constant
onslaught of connections every other minute for days on end. Does
anyone know why Googlebot just keeps crawling the FTP site
endlessly?
googlebot is an asshole!
I can certainly agree to that statement! It was just frikken
relentless. I figured it would eventually give up but days later it
was still going.
thats why my bbs telnet port is on 24 and my bbs http is on port
81
you guys could always get a router that isnt a POS
I can't, it'd a dsl modem with a router (from the isp - tds)
some people may not be able to get up and spend 150.00 dollars on a router so quickly, until then you just have to make do with what you have.
you guys could always get a router that isnt a POS
some people may not be able to get up and spend 150.00 dollars on a
router so quickly, until then you just have to make do with what
you have.
you guys could always get a router that isnt a POS
I can't, it'd a dsl modem with a router (from the isp - tds)
The best thing about that little Belkin, besides running a killer OS and being rock-solid? The price. $20 with a $20 mail in rebate. :)
I can certainly agree to that statement! It was just frikken relentless. I figured it would eventually give up but days later it was still going.
you guys could always get a router that isnt a POS
I can't, it'd a dsl modem with a router (from the isp - tds)
Re: Re: Welcome back
By: Mro to Lord Time on Fri May 15 2015 04:13 pm
you guys could always get a router that isnt a POS
some people may not be able to get up and spend 150.00 dollars on a router so quickly, until then you just have to make do with what you have.
i don't understand why you guys have such a problem with googlebot... it crawls my web sites and ftp server with no problems... sure, at one point
it may have been running a complete crawl but once that was done, it was quite well behaved and still is...
Re: Re: Welcome back
By: KenDB3 to Mickey on Fri May 15 2015 08:42 pm
So, you're saying I should take down my anarchy related junk? LOL
I think I still have the E-911 document from 1991 on my system, and some telco box instructions.
Re: Re: Welcome back
By: KenDB3 to Mro on Fri May 15 2015 08:45 pm
I can certainly agree to that statement! It was just frikken relentless. I figured it would eventually give up but days later it was still going.
There should be a flag in ROBOTS.TXT that says "if you're going to ignore this, please have the decency of only using XX number of threads"
On Fri, 15 May 2015, KenDB3 wrote to Mro:
I had to specifically block Googlebot, it was a constant
onslaught of connections every other minute for days on end. Does anyone know why Googlebot just keeps crawling the FTP site endlessly?
googlebot is an asshole!
I can certainly agree to that statement! It was just frikken
relentless. I figured it would eventually give up but days later it was still going.
i don't understand why you guys have such a problem with googlebot... it crawls my web sites and ftp server with no problems... sure, at one point it may have been running a complete crawl but once that was done, it was quite well behaved and still is...
another factor is links... if there's more than one link to a file, it will attempt to track all of them... i remember working on one site that had some sort of dynamic linking thing to all their files and pages... it made it seem that there were hundreds of pages with all the same content... all of the bots were ravenous on that site and the owners were complaining that they had no human visitors because of the bots... they blocked the bots and still didn't have a human visitors... why? because they weren't in the indexes... i went and ripped out that linking code and set plain static links to their content... then we allowed the bots back in and indexing the site was done in a very short time... much shorter than previously had been being seen... the humans followed after that... they still don't understand the problem that linking code caused... i mean they see the problem and know what it was but they don't understand how it was detrimental to them...
another thing folks can do is to set access timings in robots.txt for the various bots that recognize them... set "Crawl-delay: 300" for 5 minutes between accesses... i don't find anything specific in any of my robots.txt for googlebot, though... it may recognise it but you'll need to go look that up on the googlebot site to see for sure...
and yes, placing a robots.txt in your ftp root works... at least for googlebot... it regularly pulls mine and follows the instructions conveyed when they are merged into the index... it may take a few days or weeks but it does start following after the file has been pulled and added into the main index...
i don't understand why you guys have such a problem with
googlebot... it crawls my web sites and ftp server with no
problems... sure, at one point it may have been running a complete
crawl but once that was done, it was quite well behaved and still
is...
it shouldnt even be crawling your ftp server, though.
some people have no issues. other people are just raped non stop by googlebot. and i've logged onto them via vnc or teamviewer and seen
this.
I can certainly agree to that statement! It was just frikken relentless. I figured it would eventually give up but days later
it was still going.
i don't understand why you guys have such a problem with
googlebot... it crawls my web sites and ftp server with no
problems... sure, at one point it may have been running a complete
crawl but once that was done, it was quite well behaved and still
is...
I don't really know about other folks, but my problem was that I
had 8 whopping files in the file base at the time, and now I have
9, so either way, not much to crawl. And even though there wasn't
much content, the crawl went on for *months* before I finally
blocked it.
I wouldn't have even bothered, except that it eventually slowed
down browsing of the http site, and it slowed down the terminal
access as well and I had noticeable delays logging in, reading
messages, launching doors, etc.... But, when I turned off the FTP,
all of those delays went away. Admittedly, my sbbs runs on an older machine running XP, mainly because I don't need (or want to spend
any money on) any big hardware upgrades, because most of the time
it's perfectly fine for what I need.
I assure you, I'm not complaining because I hated seeing the
traffic, I'm complaining because I really didn't want to block it,
but had to because I didn't know how long it was going to keep it
up.
another factor is links... if there's more than one link to a file,
it will attempt to track all of them... i remember working on one
site that had some sort of dynamic linking thing to all their files
and pages... it made it seem that there were hundreds of pages with
all the same content... all of the bots were ravenous on that site
and the owners were complaining that they had no human visitors
because of the bots... they blocked the bots and still didn't have a
human visitors... why? because they weren't in the indexes... i went
and ripped out that linking code and set plain static links to their content... then we allowed the bots back in and indexing the site
was done in a very short time... much shorter than previously had
been being seen... the humans followed after that... they still
don't understand the problem that linking code caused... i mean they
see the problem and know what it was but they don't understand how
it was detrimental to them...
I wonder if the way sbbs adds some random text after 00index.html
has anything to do with the way googlebot acts. I remember reading somewhere that there was a purpose to the randomized text, but
can't remember the intended purpose.
another thing folks can do is to set access timings in robots.txt
for the various bots that recognize them... set "Crawl-delay: 300"
for 5 minutes between accesses... i don't find anything specific in
any of my robots.txt for googlebot, though... it may recognise it
but you'll need to go look that up on the googlebot site to see for
sure...
I didn't know that. I wonder if it would have helped or not. My
system was bogged down I think because googlebot was hitting the
FTP what looked like every 2 to 4 minutes (give or take).
and yes, placing a robots.txt in your ftp root works... at least for googlebot... it regularly pulls mine and follows the instructions
conveyed when they are merged into the index... it may take a few
days or weeks but it does start following after the file has been
pulled and added into the main index...
Well, food for thought. My BBS has sped up since I blocked it, and
its not like a company web site where I would certainly *want* to
be crawled, so I'm pretty happy with my decision. But, I could
always try it and open the flood gates again (but again, I'm a
happier now, so I probably won't lol).
I would really just love to know why it started on 9/12/2014 (5
times that particular day, with spread out intervals), and then the
very next day it crawled 479 times (short intervals). And then
about the same ammount every day afterwards until 4/14/2015 when I
put the block up. It crawled my 8 files for 7 months? Really?
Really Really? What caused the loop? And the even better question,
why did googlebot not detect a loop?
it shouldnt even be crawling your ftp server, though.
why not??
then there's a problem somewhere... it is well known, too, that there are spiders out there that say they are googlbot but are not... many of those
do not play nice...
it shouldnt even be crawling your ftp server, though.
why not??
because by definition googlebot is a web crawling bot that discovers "pages"
it shouldnt be on people's ftp servers. it shouldnt be using the ftp protocol.
then there's a problem somewhere... it is well known, too, that there are spiders out there that say they are googlbot but are not... many of those
do not play nice...
i never go by what the bot reports. i look at the ip addresses.
fact is, googlebot and other spiders dont always obey what we ask of
them.
Re: Re: Welcome back
By: KenDB3 to kk4qbn on Thu May 14 2015 07:34 pm
I had to specifically block Googlebot, it was a constant onslaught of connections every other minute for days on end. Does anyone know why Googlebot just keeps crawling the FTP site endlessly?
googlebot is an asshole!
I can certainly agree to that statement! It was just frikken relentless. I figured it would eventually give up but days later it was still going.
tested to see if anything works yet, I can get into it myself, I do think my firewall opened correctly, but my router is weird, I do not NAT is working because all the hack attempts I'm getting via telnet (TONS compared to the way it used to be), but if I try to login using the nwga_net.synchro.net name from the local network It goes straight to my router setup, both on port 23 and 80.
thats why my bbs telnet port is on 24 and my bbs http is on port 81
you guys could always get a router that isnt a POS
I can't, it'd a dsl modem with a router (from the isp - tds)
On Fri, 15 May 2015, KenDB3 wrote to Mro:
I had to specifically block Googlebot, it was a constant
onslaught of connections every other minute for days on end. Does anyone know why Googlebot just keeps crawling the FTP site endlessly?
googlebot is an asshole!
I can certainly agree to that statement! It was just frikken relentless. I figured it would eventually give up but days later it was still going.
i don't understand why you guys have such a problem with googlebot... it crawls my web sites and ftp server with no problems... sure, at one point it may have been running a complete crawl but once that was done, it was quite well behaved and still is...
I don't really know about other folks, but my problem was that I had 8 whopping files in the file base at the time, and now I have 9, so either way, not much to crawl. And even though there wasn't much content, the crawl went on for *months* before I finally blocked it.
I wouldn't have even bothered, except that it eventually slowed down browsing of the http site, and it slowed down the terminal access as well and I had noticeable delays logging in, reading messages, launching doors, etc.... But, when I turned off the FTP, all of those delays went away. Admittedly, my sbbs runs on an older machine running XP, mainly because I don't need (or want to spend any money on) any big hardware upgrades, because most of the time it's perfectly fine for what I need.
I assure you, I'm not complaining because I hated seeing the traffic, I'm complaining because I really didn't want to block it, but had to because I didn't know how long it was going to keep it up.
another factor is links... if there's more than one link to a file, it will attempt to track all of them... i remember working on one site that had some sort of dynamic linking thing to all their files and pages... it made it seem that there were hundreds of pages with all the same content... all of the bots were ravenous on that site and the owners were complaining that they had no human visitors because of the bots... they blocked the bots and still didn't have a human visitors... why? because they weren't in the indexes... i went and ripped out that linking code and set plain static links to their content... then we allowed the bots back in and indexing the site was done in a very short time... much shorter than previously had been being seen... the humans followed after that... they still don't understand the problem that linking code caused... i mean they see the problem and know what it was but they don't understand how it was detrimental to them...
I wonder if the way sbbs adds some random text after 00index.html has anything to do with the way googlebot acts.
I remember reading somewhere
that there was a purpose to the randomized text, but can't remember the intended purpose.
another thing folks can do is to set access timings in robots.txt for the various bots that recognize them... set "Crawl-delay: 300" for 5 minutes between accesses... i don't find anything specific in any of my robots.txt for googlebot, though... it may recognise it but you'll need to go look that up on the googlebot site to see for sure...
I didn't know that. I wonder if it would have helped or not. My system was bogged down I think because googlebot was hitting the FTP what looked like every 2 to 4 minutes (give or take).
00:00:21 1684 CTRL connection accepted from: 66.249.73.128 port 54041 00:00:21 1684 Hostname: crawl-66-249-73-128.googlebot.com
00:00:22 1684 Guest: <googlebot@google.com>
00:00:22 1684 Guest logged in (1 today, 63413 total)
00:00:22 1684 Guest downloading HTML index for / in passive mode
00:00:22 1684 Transfer successful: 3621 bytes sent in 0 seconds (7242 cps) 00:00:22 1684 Guest logged off
00:00:22 1684 CTRL thread terminated (0 clients and 1 threads remain, 872 served) 00:03:02 1736 CTRL connection accepted from: 66.249.73.128 port 54480 00:03:02 1736 Hostname: crawl-66-249-73-128.googlebot.com
00:03:02 1736 Guest: <googlebot@google.com>
00:03:02 1736 Guest logged in (2 today, 63414 total)
00:03:03 1736 Guest downloading HTML index for / in passive mode
00:03:03 1736 Transfer successful: 3621 bytes sent in 0 seconds (7242 cps) 00:03:03 1736 Guest logged off
00:03:03 1736 CTRL thread terminated (0 clients and 1 threads remain, 873 served) etc...
and yes, placing a robots.txt in your ftp root works... at least for googlebot... it regularly pulls mine and follows the instructions conveyed when they are merged into the index... it may take a few days or weeks but it does start following after the file has been pulled and added into the main index...
Well, food for thought. My BBS has sped up since I blocked it, and its not like a company web site where I would certainly *want* to be crawled, so I'm pretty happy with my decision. But, I could always try it and open the flood gates again (but again, I'm a happier now, so I probably won't lol).
I would really just love to know why it started on 9/12/2014 (5 times that particular day, with spread out intervals), and then the very next day it crawled 479 times (short intervals). And then about the same ammount every day afterwards until 4/14/2015 when I put the block up. It crawled my 8 files for 7 months? Really? Really Really? What caused the loop? And the even better question, why did googlebot not detect a loop?
because by definition googlebot is a web crawling bot that discovers "pages"
you shold tell google that, then...
it shouldnt be on people's ftp servers. it shouldnt be using the ftp protocol.
spiders and bots follow links... if a link is using the ftp protocol, they follow it just as well and easily as following a http link...
it takes time... that's what folks don't understand... they expect that
when they see GB getting their robots.txt every day that they can make a change, the bot will grab it and immediately start following it... the bot cannot and does not parse the contents of the robots.txt file... it
delivers those contents back to the database for parsing and indexing... when the master database is updated, then and only then can the new instructions in the site's robots.txt be applied and followed...
and yes, placing a robots.txt in your ftp root works... at least
for googlebot... it regularly pulls mine and follows the
instructions conveyed when they are merged into the index... it
may take a few days or weeks but it does start following after
the file has been pulled and added into the main index...
Just an FYI, FTP crawlers do not look for or adhere to any
robots.txt files. That's for HTTP crawlers only.
it takes time... that's what folks don't understand... they expect
that when they see GB getting their robots.txt every day that they
can make a change, the bot will grab it and immediately start
following it... the bot cannot and does not parse the contents of
the robots.txt file... it delivers those contents back to the
database for parsing and indexing... when the master database is
updated, then and only then can the new instructions in the site's robots.txt be applied and followed...
are you SURE, it behaves that way?
i thought it first looked for the robots.txt rules, then followed
them (but as i said it does not always do that).
the method you described sounds pretty ass-backwards.
i do know it phones home after several rejections. or it's supposed
to.
i'm pretty sure it reads the robots.txt file first thing.
it shouldnt even be crawling your ftp server, though.
some people have no issues. other people are just raped non stop by googlebot. and i've logged onto them via vnc or teamviewer and seen this.
On Tue, 19 May 2015, Digital Man wrote to KenDB3:
and yes, placing a robots.txt in your ftp root works... at least
for googlebot... it regularly pulls mine and follows the
instructions conveyed when they are merged into the index... it
may take a few days or weeks but it does start following after
the file has been pulled and added into the main index...
Just an FYI, FTP crawlers do not look for or adhere to any
robots.txt files. That's for HTTP crawlers only.
https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
Robots.txt Specifications
Abstract
Requirements Language
Basic Definitions
Applicability
File location & range of validity
2nd paragraph
Google-specific: Google also accepts and follows robots.txt files
for FTP sites. FTP-based robots.txt files are
accessed via the FTP protocol using an anonymous
login.
Google-specific: Google also accepts and follows robots.txt files
for FTP sites. FTP-based robots.txt files are
accessed via the FTP protocol using an anonymous
login.
Interesting. In my experiments, I never saw the google FTP crawler
adhere to the file, so I just ended up blocking it based on the
email address used for an anonymous-FTP password. Perhaps they
added the support for robots.txt via FTP later.
I've blocked the whole range of ip addresses for googlebot that Lord Time provided, which locked googlebot out, but it is still constantly hammering my ftp server. RELENTLESS!
I had to specifically block Googlebot, it was a constant onslaught of connections every other minute for days on end. Does anyone know why Googlebot just keeps crawling the FTP site endlessly?
Re: Re: Welcome back
By: KenDB3 to kk4qbn on Thu May 14 2015 07:34 pm
I had to specifically block Googlebot, it was a constant onslaught of
connections every other minute for days on end. Does anyone know why
Googlebot just keeps crawling the FTP site endlessly?
Yeah, I looked into it, and it was because of the random sequence appeneded by the index generation. The index has been updated to not do that anymore since, but Google will still try to crawl every random URL it has cached for a very long time (it's been many months since the fix, and my VPS is still getting over 100 queres per minute from Googlebot).
Re: Re: Welcome back
By: KenDB3 to kk4qbn on Thu May 14 2015 07:34 pm
I had to specifically block Googlebot, it was a constant onslaught of connections every other minute for days on end. Does anyone know why Googlebot just keeps crawling the FTP site endlessly?
Yeah, I looked into it, and it was because of the random sequence appeneded the index generation. The index has been updated to not do that anymore sin but Google will still try to crawl every random URL it has cached for a very long time (it's been many months since the fix, and my VPS is still getting over 100 queres per minute from Googlebot).
i never go by what the bot reports. i look at the ip addresses. fact is, googlebot and other spiders dont always obey what we ask of them.
Yeah, I looked into it, and it was because of the random sequence appeneded the index generation. The index has been updated to not do that anymore sin but Google will still try to crawl every random URL it has cached for a very long time (it's been many months since the fix, and my VPS is still getting over 100 queres per minute from Googlebot).
I kind of had a feeling. Thanks for the info! Was there a reason for the randomized appended text previously?
this is a block list of peerblock i use to control googlebot and cie
Yeah, I looked into it, and it was because of the random sequence appeneded the index generation. The index has been updated to not do that anymore sin but Google will still try to crawl every random URL it has cached for a very long time (it's been many months since the fix, and my VPS is still getting over 100 queres per minute from Googlebot).
I kind of had a feeling. Thanks for the info! Was there a reason for the randomized appended text previously?
The reason was to defeat browser caches.
| Sysop: | MCMLXXIX |
|---|---|
| Location: | Prospect, CT |
| Users: | 325 |
| Nodes: | 10 (0 / 10) |
| Uptime: | 06:18:29 |
| Calls: | 510 |
| Messages: | 220570 |