Warning: Keep out!
July 26, 2007
Thank you everyone for your words and your concerns. It is always good to hear and be reminded that not everyone out there is a scary, threatening person. Thank you!
Now:
Some of you have emailed me asking about what the hubby is doing to the site to help keep undesirables away from the blog. Here's Ryan describing what he did, in case anyone else feels the need to follow suit…
Howdy, all. Here's a quick a crash course in how to (at least partially) keep the sickos away from your site.
Angela found in her site statistics that visitors were coming from Google (and possibly other search engines) having searched for disturbing phrases. One way to keep the weirdos out, we decided, was to keep them from being able to find the site through the search engines. There are a few ways to do this.
(Here comes Ryan's technical description which may scramble your brain, if so, move on to read Option 1, which even I understood how to do.)
A search engine (SE) scours the internet by using computer programs called robots (sometimes spiders) that jump from web links to pages, add the pages to their database, then follow all of the links on those pages to other pages, which they add to the database, then follow its links on to still more pages, ad infinitum. They call this indexing. During an index, robots will add billions of pages to a database that the SE users then search. Because indexing billions of pages uses a huge amount of computing power, SEs are not interested in indexing sites with no desire to be indexed. For this reason, web developers came up with a standard called robots.txt to allow web site owners to keep parts of their sites from being visited by the robot programs.
To keep SEs from indexing your site, you need to let them know that you do not want their robot indexing your pages. There are two ways to do this. One involves adding a line to the HTML of your web pages. The other involves uploading a file to the root directory of your website. Either one tells a visiting robot that you don't want them there, and away they go without adding you to their database.
These methods can be used selectively, i.e., you can keep out ONLY Google's robot, or turn robots away from only certain files. We're more interested in getting the entire site out of every SE index, so that's what these directions are for.
OPTION 1:
If you don't have the ability to upload files to the main directory, but you do use a blogging program with templates (such as blogger or wordpress), you can add a line to your site template that will then be reproduced on every page of your website and will bounce robots away when they try to visit.
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW" />
This line is a "meta tag," something that is read by browsers and robots, but does not appear on the visitor's screen. When a robot sees this, it leaves without indexing. Using whatever method you have to alter your template, add the above line inside of the <head> tag of your template.
The <head> tag is near the top of the web page code. If you look for the line your template that says…
<head>
…then paste in the meta tag just below that, it will be in the right place. Be careful not to put it after the line that says..
</head>
… This is the head closing tag. You want the opening one without the forward slash.
OPTION 2:
If you are able to upload files to the main directory of your web site, using a robots.txt file is the best choice. This is a text file that all search engines look for when first coming to a site. If the robots.txt file contains instructions for robots to go away, they do. To turn them all away, all the time, the robots.txt file should contain this text, exactly like this:
User-agent: *
Disallow: /
You have to upload this to the main directory of your site or it won't block robots from everything. Here's a downloadable copy of the file that you can use.
NEXT STEP
According to search engine folks, you now have to wait. Eventually, SE robots will visit your site as they crawl the internet, see that you are rejecting them, and will move on. As your site continues to remain inaccessible, the pages already in their index will drop out. Eventually, your site will be gone from the search engine's database. It will still be available to any visitors that know the address, as well as through any links that other sites or bloggers have made to your site. The people searching on Google, however, will not.
Google allows web site owners to take an additional step and request the removal of their site from the index. You have to set up one or both of the above options first, then you can request removal. We did this yesterday and the request is still pending. I have not been able to find an option like this with other search engines, and with Google's you have to register with them as a webmaster. Theoretically, this step is unnecessary. The robots.txt thing is supposed to (eventually) ensure your disappearance from searchable web space, we're just not sure how long it takes.
ADDED: In order to request removal, you need to sign up with Google's webmaster tools. The sign up page is here. Once you've signed up, you can click on Webmaster Tools, then your site name, then URL Removals. The request form is there. note, however, that this is not strictly necessary. Blocking the robot is enough to get you out of the index. In Angela's case, the site was dropped before Google even got around to noticing the request for removal.
NOTE: It took about three days from doing all this for Google to drop Colorfool from the index.
If you feel the need to safeguard your blog and family like this and you need help or have any questions, email Angela and we'll walk you through it.
If you want detailed information, see http://www.robotstxt.org/wc/robots.html.
Did ya get all that? You can't find me through google searches anymore. Which is wonderful for me. However, if you are a site that sells something I'm sure you want people to find you. Since we made this change, traffic to my site has been halved-just keep that in mind if you are trying to make money through your blog. Everyone who reads your blog currently will still be able to read your blog using the same method they always have-unless they always google search you. Just give everyone a heads up on your post and tell them to bookmark you or add your web address to a bloglines type thingie.
Also, if you use FlickR you can safeguard your photos by making them only viewable by friends and family and if you go to "your account" and then the "privacy and permissions" sections you can make you photos unsearchable there and a few other things too, if you feel the need. If you are titling & tagging your photos- be safe. I'd stay away from photo titles/labels like "little girl" "small children" and well, just don't identify that you have photos of your babes if you are not removing yourself from searches.
I hope this is helpful and not too confusing. If you are confused, try reading it again. I had too! I'm dense when it comes to tech. stuff. Good luck all.
Filed under: random chatter
11 Comments Leave a Comment
1.
martha | July 26, 2007 at 9:17 am
Thanks so much for the info. I enjoy reading your blog so much. I am sorry this happened to you, but I’m glad you have a hubby who could fix it. Take care of yourself!
2.
gloria | July 26, 2007 at 11:03 am
THANK YOU
THANK YOU
THANK YOU
for sharing this info…i typed it in just as you had it, so hopefully i did it correctly.
this is one of the reasons i never really put my babie’s names up on my blog, or share too many of my scrapbook pages with info on them, which, like you, made me wonder what the blog was really for, ya know?
anyhow.
THANKS.
3.
capello | July 26, 2007 at 11:47 am
dude. what the fuck were they googling to find you? you’re so damn “clean” i can’t figure out how naughty people found you.
4.
Ellie | July 26, 2007 at 11:49 am
Oh, thank you thank you thank you. That was THE easiest web instruction thing-o I have ever read. I hope I did it correctly…. I did wonder why I started a blog + flickr, as I’m so paranoid about my kids “being out there”. I hope this works. Also, while you’re on the subject of web tutorials – would you (or hubby) be able to tell me how to find site statistics on my blog. I’m also curious about who is visiting. (I had a weirdo asking to add me to their flickr favourites, which has made me even more careful).
PS – I love your blog, little snippets of your day. I hope we can all continue, as a community around the world, to be able to share so openly and easily. And not have to worry about those horrible parts of this internet thing.
Thanks.
5.
Robyn | July 26, 2007 at 1:48 pm
this is great! thank you so much for showing us how to do this.
6.
sue | July 26, 2007 at 5:50 pm
Angela~
I just tried to google ‘colorfool’ on google. Your current site didn’t come up, but your old site did (colorfool creates), which had a link in the last post to your new site. You’re probably already working on it- but just in case, I thought I’d let you know.
And thanks so much for all the effort/time you guys put into this last post. If I can wrap my little brain around it (seeing as I am the tech savvy one in our household. This is not saying much;) I am going to try and do this on our site.
Hope all is otherwise well. Love the blog & congrats on ‘Baby Boy!’
7.
Amanda | July 26, 2007 at 10:15 pm
Thanks for taking the time to explain this to someone who has no clue.
Your advice is much appreciated!
8.
Bex | July 27, 2007 at 8:52 am
I’ve been thinking about this a lot lately but not really knowing what to do about it all.
Our family blogs to keep in touch with other family members far away. Through doing this we have been lucky enough to make some lovely new friends also.
Is it wrong of me not to want to know who has visited us anon. before we protected ourselves? I don’t think I really want to know…
Thank-you for helping to keep us all safe. Xx
9.
Angela A. | July 29, 2007 at 2:50 pm
Thanks so much for the very useful info!!! As a new blogger this will help me protect my wee ones too! I have enjoyed your site for sometime – I am a bad commentor – very lazy sorry! Thanks again and I hope the rest of your summer is full of warm sunny days full of happy memories!
10.
cindy | July 31, 2007 at 11:56 am
thanks for the info! you always think things will never happen to you, until they do.
11.
Amy | August 13, 2007 at 12:53 pm
Thanks so much for posting this. Very generous of you to spend your time delineating it all. This is very very helpful!
Leave a Comment
XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
TrackBack URL | RSS feed for comments on this post.