Duskrider's Blog

Well, back to ye olde website design grind I suppose. That other little project I alluded to in my last post is eating up a fair bit of my free time. It's funny how things start to 'pop up' when you're getting close to the end of a project. Little bugs and strange coding things somehow work their way into your project and you have to go through and do your best to wipe them out.

The real problem is that I've been away from it for so long that I forgot about some of the important backend things that every website should have... and nobody ever sees. So, because I always forget these things, and because I always dread writing them when I finally remember them... I've decided to code them in such a way that I'll be able to just plug 'em into any site I make in the future, as well as all the sites I currently have.

#1. The bad bot blocker / banner.
Very few people realize just how much traffic there is on the internet that isn't human. It's pathetic, scary, and a little demoralizing. If you'll notice, on my blog I don't allow comments, trackbacks, or any other user submitted content. Know why? It's not because I don't value your opinion. It's because the only 'visitors' that ever commented on my blog entries were bots. Robots. Computer Programs. They crawl the web, looking for nice juicy ways of mucking everything up with no regard for anything but their master's will or profit. They would come into this blog and write comments, with links of course, about how great viagra is, or how you can get cialis very cheap by clicking HERE. If I hadn't put a comment block on there in the first place, you wouldn't even be able to read this post for the number of bot comments there would be. It used to be that I would allow comments, but only after I approved them personally. When it was like that, I would get about 10-50 e-mails a day about comments being made to my blog. EVERY DAY, and the blog was only a month or two old. Out of all that, only one comment was legitimate. So, to get rid of the spam to my inbox I just disabled comments.

The bots still visit. Every day. They just can't do anything.

I want to change that. So, I'm in the process of writing a script that will weed out bad bots, identify them, allow me to see when they come into my site and how often, and then BAN them from ever visiting my site again. Ahh... to be fresh again.

#2. The Captcha Script
This one is a beauty. It goes hand in hand with number one. Bots can be pretty smart, but they're never going to be as smart as a human (well, we can hope). I'm sure you've probably seen a Captcha Script before and never even knew what it was all about.

A Captcha script creates a graphic with letters and numbers contained inside that is impossible (or very very hard) for a computer program to read, but usually pretty easy for a human to read. If you've ever signed up for something and had to "Enter the letters or numbers you see into the box then click submit," you've used a Captcha script. Bots can't read that. It's extremely easy for a bot to read text like what I'm typing right now, but if I were to put it inside a graphic file... the only way to read that is visually. Bots don't have eyes. This makes sure that whatever content is beyond the Captcha script is only being accessed by a human. I love Captcha scripts.

I looked all over the internet for one to do things just the way I wanted... and couldn't find one. So, I wrote my own. I've used it a couple of times already, though I can't give you a link to check it out just yet.

#3. Sitemap Script
As I've written before, search engines like Google and Yahoo are necessary evils. There's just no getting around it, you need to be listed to get traffic. Google, and more recently Yahoo, MSN, and some of the other big search names, have started accepting SiteMaps from websites in order to facilitate better indexing. Basically a sitemap is a... well... map of a website. It lists all the files, directories, and structures as well as how often pages are updated and some other useful information about the site. Google et all then use this information as a basis for checking your site.

It makes things easier on them because now they don't have to guess about your site structure... they can just go check it out. Anything that makes life for the Search Engines easier is good, at least when it concerns my websites, so a sitemap is a must. They are, however, a pain to create. Each engine wants them in a different format, XML, RSS, Text file, you name it. There are a plethora of tools out there to create and upload them to the search engine manually... but I want things automatic. Hands off. Plug it in and let it run. On and on, etc, etc. So, to that end, I'll just write my own. That way I can have it run automagically once a day, create a sitemap file in several different formats, notify the search engines that my new sitemap is available, and go along on my merry way knowing that each and every day the search engines are being fed the latest and greatest information about my site. B-E-A-utiful.

I love automagic. All of these scripts, once written, will be zipped into a nice little care package that I can just throw into the root directory of my new site. There, with a bit of modification, they will blossom and do that voodoo that they do so well. Protecting me, promoting me, and allowing me to do more important things with my time than manually block bots or send updates to Google... like play golf.

-Ed